Skip to contentSkip to conversation

kvm: Make ioeventfd usable on s390.

RFC PATCH Superseded Cornelia Huck 2 participants 14 messages Feb 22, 2013 – Feb 25, 2013

Subsystems (4)

v2 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v2 cover kvm: Make ioeventfd usable on s390.

Here's the second attempt at implementing ioeventfd for s390.

Rather than the architecture-specific functions used in v1, we now try to integrate with the kvm_io_device infrastructure. Calls to diagnose 500 subcode 3 are now mapped to _write. These devices are created on a new KVM_CSS_BUS when using a new flag KVM_IOEVENTFD_FLAG_CSS. addr and datamatch are (ab)used to contain the subchannel id and the virtqueue.

A drawback is that this interface is not easily extendable should we want to attach other hypercalls or carry more payload.

Another limitation is the limit of 1000 io devices per bus, which we would hit easily with a few hundred devices, but that should be fixable.

v1 -> v2:
- Move irqfd initialization from a module init function to kvm_init, eliminating the need for a second module for kvm/s390.
- Use kvm_io_device for s390 css devices.

Cornelia Huck (4):
KVM: Initialize irqfd from kvm_init().
KVM: Introduce KVM_CSS_BUS.
KVM: ioeventfd for s390 css devices.
KVM: s390: Wire up ioeventfd.

 Documentation/virtual/kvm/api.txt |  7 +++++++
 arch/s390/kvm/Kconfig             |  1 +
 arch/s390/kvm/Makefile            |  2 +-
 arch/s390/kvm/diag.c              | 25 +++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c          |  1 +
 include/linux/kvm_host.h          | 14 ++++++++++++++
 include/uapi/linux/kvm.h          |  2 ++
 virt/kvm/eventfd.c                | 15 ++++++++-------
 virt/kvm/kvm_main.c               |  6 ++++++
 9 files changed, 65 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.12.4
#1 lore raw

RFC PATCH v2 1/4 KVM: Initialize irqfd from kvm_init().

Currently, eventfd introduces module_init/module_exit functions to initialize/cleanup the irqfd workqueue. This only works, however, if no other module_init/module_exit functions are built into the same module.

Let's just move the initialization and cleanup to kvm_init and kvm_exit. This way, it is also clearer where kvm startup may fail.

---
 include/linux/kvm_host.h | 13 +++++++++++++
 virt/kvm/eventfd.c       |  7 ++-----
 virt/kvm/kvm_main.c      |  6 ++++++
 3 files changed, 21 insertions(+), 5 deletions(-)
include/linux/kvm_host.h +13 -0
index 722cae7..3b768ef 100644
@@ -423,6 +423,19 @@ void kvm_vcpu_uninit(struct kvm_vcpu *vcpu);
 int __must_check vcpu_load(struct kvm_vcpu *vcpu);
 void vcpu_put(struct kvm_vcpu *vcpu);
 
+#ifdef __KVM_HAVE_IOAPIC
+int kvm_irqfd_init(void);
+void kvm_irqfd_exit(void);
+#else
+static inline int kvm_irqfd_init(void)
+{
+	return 0;
+}
+
+static inline void kvm_irqfd_exit(void)
+{
+}
+#endif
 int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 		  struct module *module);
 void kvm_exit(void);
virt/kvm/eventfd.c +2 -5
index b6eea5c..f0ced1a 100644
@@ -544,7 +544,7 @@ void kvm_irq_routing_update(struct kvm *kvm,
  * aggregated from all vm* instances. We need our own isolated single-thread
  * queue to prevent deadlock against flushing the normal work-queue.
  */
-static int __init irqfd_module_init(void)
+int kvm_irqfd_init(void)
 {
 	irqfd_cleanup_wq = create_singlethread_workqueue("kvm-irqfd-cleanup");
 	if (!irqfd_cleanup_wq)
@@ -553,13 +553,10 @@ static int __init irqfd_module_init(void)
 	return 0;
 }
 
-static void __exit irqfd_module_exit(void)
+void kvm_irqfd_exit(void)
 {
 	destroy_workqueue(irqfd_cleanup_wq);
 }
-
-module_init(irqfd_module_init);
-module_exit(irqfd_module_exit);
 #endif
 
 /*
virt/kvm/kvm_main.c +6 -0
index adc68fe..7c188a3 100644
@@ -2920,6 +2920,9 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,
 	int r;
 	int cpu;
 
+	r = kvm_irqfd_init();
+	if (r)
+		goto out_irqfd;
 	r = kvm_arch_init(opaque);
 	if (r)
 		goto out_fail;
@@ -3000,6 +3003,8 @@ out_free_0a:
 out_free_0:
 	kvm_arch_exit();
 out_fail:
+	kvm_irqfd_exit();
+out_irqfd:
 	return r;
 }
 EXPORT_SYMBOL_GPL(kvm_init);
@@ -3016,6 +3021,7 @@ void kvm_exit(void)
 	on_each_cpu(hardware_disable_nolock, NULL, 1);
 	kvm_arch_hardware_unsetup();
 	kvm_arch_exit();
+	kvm_irqfd_exit();
 	free_cpumask_var(cpus_hardware_enabled);
 }
 EXPORT_SYMBOL_GPL(kvm_exit);
Signature · 1 line
1.7.12.4
#2 lore raw

RFC PATCH v2 2/4 KVM: Introduce KVM_CSS_BUS.

In reply to Cornelia Huck

Add a new bus type for s390 css kvm io devices.

---
 include/linux/kvm_host.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/kvm_host.h +1 -0
index 3b768ef..59be516 100644
@@ -148,6 +148,7 @@ struct kvm_io_bus {
 enum kvm_bus {
 	KVM_MMIO_BUS,
 	KVM_PIO_BUS,
+	KVM_CSS_BUS,
 	KVM_NR_BUSES
 };
 
Signature · 1 line
1.7.12.4
#3 lore raw

Re: [RFC PATCH v2 2/4] KVM: Introduce KVM_CSS_BUS.

Quoted 12 lines · Cornelia Huck

Add a new bus type for s390 css kvm io devices.

---
 include/linux/kvm_host.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/kvm_host.h +0 -0
index 3b768ef..59be516 100644

include/linux/kvm_host.h struct kvm_io_bus { · line 151

Quoted 4 lines
include/linux/kvm_host.h +1 -0
@@ -148,6 +148,7 @@ struct kvm_io_bus {
 enum kvm_bus {
 	KVM_MMIO_BUS,
 	KVM_PIO_BUS,
+	KVM_CSS_BUS,

so maybe KVM_S390_VIRTIO_CCW_NOTIFY_BUS

to make the fact it's s390 and virtio specific, explicit?

Quoted 5 lines
include/linux/kvm_host.h +0 -0
 	KVM_NR_BUSES
 };
 
Signature · 1 line
1.7.12.4
#4 lore raw

Re: [RFC PATCH v2 2/4] KVM: Introduce KVM_CSS_BUS.

Quoted 20 lines · On Sun, 24 Feb 2013 11:57:25 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 12 lines · Cornelia Huck

Add a new bus type for s390 css kvm io devices.

---
 include/linux/kvm_host.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/kvm_host.h +0 -0
index 3b768ef..59be516 100644

include/linux/kvm_host.h struct kvm_io_bus { · line 151

Quoted 4 lines
include/linux/kvm_host.h +1 -0
@@ -148,6 +148,7 @@ struct kvm_io_bus {
 enum kvm_bus {
 	KVM_MMIO_BUS,
 	KVM_PIO_BUS,
+	KVM_CSS_BUS,

so maybe KVM_S390_VIRTIO_CCW_NOTIFY_BUS

Quoted 1 line

to make the fact it's s390 and virtio specific, explicit?

Hm, I don't really see a need to point out that this is s390-specific, so I think I'd prefer KVM_VIRTIO_CCW_NOTIFY_BUS.

Quoted 7 lines
Quoted 6 lines
include/linux/kvm_host.h +0 -0
 	KVM_NR_BUSES
 };
 
Signature · 1 line
1.7.12.4
#5 lore raw

RFC PATCH v2 3/4 KVM: ioeventfd for s390 css devices.

In reply to Cornelia Huck

Enhance KVM_IOEVENTFD with a new flag that allows to attach to s390 css devices.

---
 Documentation/virtual/kvm/api.txt | 7 +++++++
 include/uapi/linux/kvm.h          | 2 ++
 virt/kvm/eventfd.c                | 8 ++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)
Documentation/virtual/kvm/api.txt +7 -0
index c2534c3..40e799c 100644
@@ -1468,15 +1468,22 @@ struct kvm_ioeventfd {
 	__u8  pad[36];
 };
 
+For the special case of s390 css devices, the ioevent is matched to a
+subchannel/virtqueue tuple instead.
+
 The following flags are defined:
 
 #define KVM_IOEVENTFD_FLAG_DATAMATCH (1 << kvm_ioeventfd_flag_nr_datamatch)
 #define KVM_IOEVENTFD_FLAG_PIO       (1 << kvm_ioeventfd_flag_nr_pio)
 #define KVM_IOEVENTFD_FLAG_DEASSIGN  (1 << kvm_ioeventfd_flag_nr_deassign)
+#define KVM_IOEVENTFD_FLAG_CSS       (1 << kvm_ioeventfd_flag_nr_css)
 
 If datamatch flag is set, the event will be signaled only if the written value
 to the registered address is equal to datamatch in struct kvm_ioeventfd.
 
+For css devices, addr contains the subchannel id and datamatch the virtqueue
+index.
+
 
 4.60 KVM_DIRTY_TLB
 
include/uapi/linux/kvm.h +2 -0
index 9a2db57..1df0766 100644
@@ -448,12 +448,14 @@ enum {
 	kvm_ioeventfd_flag_nr_datamatch,
 	kvm_ioeventfd_flag_nr_pio,
 	kvm_ioeventfd_flag_nr_deassign,
+	kvm_ioeventfd_flag_nr_css,
 	kvm_ioeventfd_flag_nr_max,
 };
 
 #define KVM_IOEVENTFD_FLAG_DATAMATCH (1 << kvm_ioeventfd_flag_nr_datamatch)
 #define KVM_IOEVENTFD_FLAG_PIO       (1 << kvm_ioeventfd_flag_nr_pio)
 #define KVM_IOEVENTFD_FLAG_DEASSIGN  (1 << kvm_ioeventfd_flag_nr_deassign)
+#define KVM_IOEVENTFD_FLAG_CSS       (1 << kvm_ioeventfd_flag_nr_css)
 
 #define KVM_IOEVENTFD_VALID_FLAG_MASK  ((1 << kvm_ioeventfd_flag_nr_max) - 1)
 
virt/kvm/eventfd.c +6 -2
index f0ced1a..7347652 100644
@@ -679,11 +679,13 @@ static int
 kvm_assign_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
 {
 	int                       pio = args->flags & KVM_IOEVENTFD_FLAG_PIO;
-	enum kvm_bus              bus_idx = pio ? KVM_PIO_BUS : KVM_MMIO_BUS;
+	int                       css = args->flags & KVM_IOEVENTFD_FLAG_CSS;
+	enum kvm_bus              bus_idx;
 	struct _ioeventfd        *p;
 	struct eventfd_ctx       *eventfd;
 	int                       ret;
 
+	bus_idx = pio ? KVM_PIO_BUS : css ? KVM_CSS_BUS: KVM_MMIO_BUS;
 	/* must be natural-word sized */
 	switch (args->len) {
 	case 1:
@@ -759,11 +761,13 @@ static int
 kvm_deassign_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
 {
 	int                       pio = args->flags & KVM_IOEVENTFD_FLAG_PIO;
-	enum kvm_bus              bus_idx = pio ? KVM_PIO_BUS : KVM_MMIO_BUS;
+	int                       css = args->flags & KVM_IOEVENTFD_FLAG_CSS;
+	enum kvm_bus              bus_idx;
 	struct _ioeventfd        *p, *tmp;
 	struct eventfd_ctx       *eventfd;
 	int                       ret = -ENOENT;
 
+	bus_idx = pio ? KVM_PIO_BUS : css ? KVM_CSS_BUS: KVM_MMIO_BUS;
 	eventfd = eventfd_ctx_fdget(args->fd);
 	if (IS_ERR(eventfd))
 		return PTR_ERR(eventfd);
Signature · 1 line
1.7.12.4
#6 lore raw

Re: [RFC PATCH v2 3/4] KVM: ioeventfd for s390 css devices.

Quoted 43 lines · Cornelia Huck

Enhance KVM_IOEVENTFD with a new flag that allows to attach to s390 css devices.

---
 Documentation/virtual/kvm/api.txt | 7 +++++++
 include/uapi/linux/kvm.h          | 2 ++
 virt/kvm/eventfd.c                | 8 ++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)
Documentation/virtual/kvm/api.txt +7 -0
index c2534c3..40e799c 100644
@@ -1468,15 +1468,22 @@ struct kvm_ioeventfd {
 	__u8  pad[36];
 };
 
+For the special case of s390 css devices, the ioevent is matched to a
+subchannel/virtqueue tuple instead.
+
 The following flags are defined:
 
 #define KVM_IOEVENTFD_FLAG_DATAMATCH (1 << kvm_ioeventfd_flag_nr_datamatch)
 #define KVM_IOEVENTFD_FLAG_PIO       (1 << kvm_ioeventfd_flag_nr_pio)
 #define KVM_IOEVENTFD_FLAG_DEASSIGN  (1 << kvm_ioeventfd_flag_nr_deassign)
+#define KVM_IOEVENTFD_FLAG_CSS       (1 << kvm_ioeventfd_flag_nr_css)
 
 If datamatch flag is set, the event will be signaled only if the written value
 to the registered address is equal to datamatch in struct kvm_ioeventfd.
 
+For css devices, addr contains the subchannel id and datamatch the virtqueue
+index.
+
 
 4.60 KVM_DIRTY_TLB
 
include/uapi/linux/kvm.h +0 -0
index 9a2db57..1df0766 100644
@@ -448,12 +448,14 @@ enum {
 	kvm_ioeventfd_flag_nr_datamatch,

include/uapi/linux/kvm.h enum { · line 458

Quoted 10 lines
include/uapi/linux/kvm.h +2 -0
 	kvm_ioeventfd_flag_nr_pio,
 	kvm_ioeventfd_flag_nr_deassign,
+	kvm_ioeventfd_flag_nr_css,
 	kvm_ioeventfd_flag_nr_max,
 };
 
 #define KVM_IOEVENTFD_FLAG_DATAMATCH (1 << kvm_ioeventfd_flag_nr_datamatch)
 #define KVM_IOEVENTFD_FLAG_PIO       (1 << kvm_ioeventfd_flag_nr_pio)
 #define KVM_IOEVENTFD_FLAG_DEASSIGN  (1 << kvm_ioeventfd_flag_nr_deassign)
+#define KVM_IOEVENTFD_FLAG_CSS       (1 << kvm_ioeventfd_flag_nr_css)

So let's explicitly name is KVM_IOEVENTFD_S390_VIRTIO_CCW or even KVM_IOEVENTFD_S390_VIRTIO_CCW_NOTIFY?
Also, maybe we want to add KVM_IOEVENTFD_S390_VIRTIO_NOTIFY as well?

Quoted 39 lines
include/uapi/linux/kvm.h +0 -0
 #define KVM_IOEVENTFD_VALID_FLAG_MASK  ((1 << kvm_ioeventfd_flag_nr_max) - 1)
 
virt/kvm/eventfd.c +6 -2
index f0ced1a..7347652 100644
@@ -679,11 +679,13 @@ static int
 kvm_assign_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
 {
 	int                       pio = args->flags & KVM_IOEVENTFD_FLAG_PIO;
-	enum kvm_bus              bus_idx = pio ? KVM_PIO_BUS : KVM_MMIO_BUS;
+	int                       css = args->flags & KVM_IOEVENTFD_FLAG_CSS;
+	enum kvm_bus              bus_idx;
 	struct _ioeventfd        *p;
 	struct eventfd_ctx       *eventfd;
 	int                       ret;
 
+	bus_idx = pio ? KVM_PIO_BUS : css ? KVM_CSS_BUS: KVM_MMIO_BUS;
 	/* must be natural-word sized */
 	switch (args->len) {
 	case 1:
@@ -759,11 +761,13 @@ static int
 kvm_deassign_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
 {
 	int                       pio = args->flags & KVM_IOEVENTFD_FLAG_PIO;
-	enum kvm_bus              bus_idx = pio ? KVM_PIO_BUS : KVM_MMIO_BUS;
+	int                       css = args->flags & KVM_IOEVENTFD_FLAG_CSS;
+	enum kvm_bus              bus_idx;
 	struct _ioeventfd        *p, *tmp;
 	struct eventfd_ctx       *eventfd;
 	int                       ret = -ENOENT;
 
+	bus_idx = pio ? KVM_PIO_BUS : css ? KVM_CSS_BUS: KVM_MMIO_BUS;
 	eventfd = eventfd_ctx_fdget(args->fd);
 	if (IS_ERR(eventfd))
 		return PTR_ERR(eventfd);
Signature · 1 line
1.7.12.4
#7 lore raw

Re: [RFC PATCH v2 3/4] KVM: ioeventfd for s390 css devices.

Quoted 50 lines · On Sun, 24 Feb 2013 11:47:50 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 49 lines · Cornelia Huck

Enhance KVM_IOEVENTFD with a new flag that allows to attach to s390 css devices.

---
 Documentation/virtual/kvm/api.txt | 7 +++++++
 include/uapi/linux/kvm.h          | 2 ++
 virt/kvm/eventfd.c                | 8 ++++++--
 3 files changed, 15 insertions(+), 2 deletions(-)
Documentation/virtual/kvm/api.txt +7 -0
index c2534c3..40e799c 100644
@@ -1468,15 +1468,22 @@ struct kvm_ioeventfd {
 	__u8  pad[36];
 };
 
+For the special case of s390 css devices, the ioevent is matched to a
+subchannel/virtqueue tuple instead.
+
 The following flags are defined:
 
 #define KVM_IOEVENTFD_FLAG_DATAMATCH (1 << kvm_ioeventfd_flag_nr_datamatch)
 #define KVM_IOEVENTFD_FLAG_PIO       (1 << kvm_ioeventfd_flag_nr_pio)
 #define KVM_IOEVENTFD_FLAG_DEASSIGN  (1 << kvm_ioeventfd_flag_nr_deassign)
+#define KVM_IOEVENTFD_FLAG_CSS       (1 << kvm_ioeventfd_flag_nr_css)
 
 If datamatch flag is set, the event will be signaled only if the written value
 to the registered address is equal to datamatch in struct kvm_ioeventfd.
 
+For css devices, addr contains the subchannel id and datamatch the virtqueue
+index.
+
 
 4.60 KVM_DIRTY_TLB
 
include/uapi/linux/kvm.h +1 -0
index 9a2db57..1df0766 100644
@@ -448,12 +448,14 @@ enum {
 	kvm_ioeventfd_flag_nr_datamatch,
 	kvm_ioeventfd_flag_nr_pio,
 	kvm_ioeventfd_flag_nr_deassign,
+	kvm_ioeventfd_flag_nr_css,
 	kvm_ioeventfd_flag_nr_max,
 };
 
Quoted 7 lines

include/uapi/linux/kvm.h enum { · line 458

Quoted 4 lines
include/uapi/linux/kvm.h +1 -0
 #define KVM_IOEVENTFD_FLAG_DATAMATCH (1 << kvm_ioeventfd_flag_nr_datamatch)
 #define KVM_IOEVENTFD_FLAG_PIO       (1 << kvm_ioeventfd_flag_nr_pio)
 #define KVM_IOEVENTFD_FLAG_DEASSIGN  (1 << kvm_ioeventfd_flag_nr_deassign)
+#define KVM_IOEVENTFD_FLAG_CSS       (1 << kvm_ioeventfd_flag_nr_css)

So let's explicitly name is KVM_IOEVENTFD_S390_VIRTIO_CCW or even KVM_IOEVENTFD_S390_VIRTIO_CCW_NOTIFY?

How about KVM_IOEVENTFD_FLAG_VIRTIO_CCW_NOTIFY?

Quoted 1 line

Also, maybe we want to add KVM_IOEVENTFD_S390_VIRTIO_NOTIFY as well?

No, since s390-virtio is a dead end.

Quoted 41 lines
Quoted 40 lines
include/uapi/linux/kvm.h +0 -0
 #define KVM_IOEVENTFD_VALID_FLAG_MASK  ((1 << kvm_ioeventfd_flag_nr_max) - 1)
 
virt/kvm/eventfd.c +6 -2
index f0ced1a..7347652 100644
@@ -679,11 +679,13 @@ static int
 kvm_assign_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
 {
 	int                       pio = args->flags & KVM_IOEVENTFD_FLAG_PIO;
-	enum kvm_bus              bus_idx = pio ? KVM_PIO_BUS : KVM_MMIO_BUS;
+	int                       css = args->flags & KVM_IOEVENTFD_FLAG_CSS;
+	enum kvm_bus              bus_idx;
 	struct _ioeventfd        *p;
 	struct eventfd_ctx       *eventfd;
 	int                       ret;
 
+	bus_idx = pio ? KVM_PIO_BUS : css ? KVM_CSS_BUS: KVM_MMIO_BUS;
 	/* must be natural-word sized */
 	switch (args->len) {
 	case 1:
@@ -759,11 +761,13 @@ static int
 kvm_deassign_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
 {
 	int                       pio = args->flags & KVM_IOEVENTFD_FLAG_PIO;
-	enum kvm_bus              bus_idx = pio ? KVM_PIO_BUS : KVM_MMIO_BUS;
+	int                       css = args->flags & KVM_IOEVENTFD_FLAG_CSS;
+	enum kvm_bus              bus_idx;
 	struct _ioeventfd        *p, *tmp;
 	struct eventfd_ctx       *eventfd;
 	int                       ret = -ENOENT;
 
+	bus_idx = pio ? KVM_PIO_BUS : css ? KVM_CSS_BUS: KVM_MMIO_BUS;
 	eventfd = eventfd_ctx_fdget(args->fd);
 	if (IS_ERR(eventfd))
 		return PTR_ERR(eventfd);
Signature · 1 line
1.7.12.4
#8 lore raw

RFC PATCH v2 4/4 KVM: s390: Wire up ioeventfd.

In reply to Cornelia Huck

Enable ioeventfd support on s390 and hook up diagnose 500 virtio-ccw notifications.

---
 arch/s390/kvm/Kconfig    |  1 +
 arch/s390/kvm/Makefile   |  2 +-
 arch/s390/kvm/diag.c     | 25 +++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c |  1 +
 4 files changed, 28 insertions(+), 1 deletion(-)
arch/s390/kvm/Kconfig +1 -0
index b58dd86..3c43e30 100644
@@ -22,6 +22,7 @@ config KVM
 	select PREEMPT_NOTIFIERS
 	select ANON_INODES
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
+	select HAVE_KVM_EVENTFD
 	---help---
 	  Support hosting paravirtualized guest machines using the SIE
 	  virtualization capability on the mainframe. This should work
arch/s390/kvm/Makefile +1 -1
index 3975722..8fe9d65 100644
@@ -6,7 +6,7 @@
 # it under the terms of the GNU General Public License (version 2 only)
 # as published by the Free Software Foundation.
 
-common-objs = $(addprefix ../../../virt/kvm/, kvm_main.o)
+common-objs = $(addprefix ../../../virt/kvm/, kvm_main.o eventfd.o)
 
 ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 
arch/s390/kvm/diag.c +25 -0
index a390687..13cf74a 100644
@@ -104,6 +104,29 @@ static int __diag_ipl_functions(struct kvm_vcpu *vcpu)
 	return -EREMOTE;
 }
 
+static int __diag_virtio_hypercall(struct kvm_vcpu *vcpu)
+{
+	int ret, idx;
+
+	/* No virtio-ccw notification? Get out quickly. */
+	if (!vcpu->kvm->arch.css_support ||
+	    (vcpu->run->s.regs.gprs[1] != 3))
+		return -EOPNOTSUPP;
+
+	idx = srcu_read_lock(&vcpu->kvm->srcu);
+	/*
+	 * The layout is as follows:
+	 * - gpr 2 contains the subchannel id (passed as addr)
+	 * - gpr 3 contains the virtqueue index (passed as datamatch)
+	 */
+	ret = kvm_io_bus_write(vcpu->kvm, KVM_CSS_BUS,
+				vcpu->run->s.regs.gprs[2],
+				8, &vcpu->run->s.regs.gprs[3]);
+	srcu_read_unlock(&vcpu->kvm->srcu, idx);
+	/* kvm_io_bus_write returns -EOPNOTSUPP if it found no match. */
+	return ret;
+}
+
 int kvm_s390_handle_diag(struct kvm_vcpu *vcpu)
 {
 	int code = (vcpu->arch.sie_block->ipb & 0xfff0000) >> 16;
@@ -118,6 +141,8 @@ int kvm_s390_handle_diag(struct kvm_vcpu *vcpu)
 		return __diag_time_slice_end_directed(vcpu);
 	case 0x308:
 		return __diag_ipl_functions(vcpu);
+	case 0x500:
+		return __diag_virtio_hypercall(vcpu);
 	default:
 		return -EOPNOTSUPP;
 	}
arch/s390/kvm/kvm-s390.c +1 -0
index f822d36..04d2454 100644
@@ -142,6 +142,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_ONE_REG:
 	case KVM_CAP_ENABLE_CAP:
 	case KVM_CAP_S390_CSS_SUPPORT:
+	case KVM_CAP_IOEVENTFD:
 		r = 1;
 		break;
 	case KVM_CAP_NR_VCPUS:
Signature · 1 line
1.7.12.4
#9 lore raw

Re: [RFC PATCH v2 4/4] KVM: s390: Wire up ioeventfd.

Quoted 42 lines · Cornelia Huck

Enable ioeventfd support on s390 and hook up diagnose 500 virtio-ccw notifications.

---
 arch/s390/kvm/Kconfig    |  1 +
 arch/s390/kvm/Makefile   |  2 +-
 arch/s390/kvm/diag.c     | 25 +++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c |  1 +
 4 files changed, 28 insertions(+), 1 deletion(-)
arch/s390/kvm/Kconfig +1 -0
index b58dd86..3c43e30 100644
@@ -22,6 +22,7 @@ config KVM
 	select PREEMPT_NOTIFIERS
 	select ANON_INODES
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
+	select HAVE_KVM_EVENTFD
 	---help---
 	  Support hosting paravirtualized guest machines using the SIE
 	  virtualization capability on the mainframe. This should work
arch/s390/kvm/Makefile +1 -1
index 3975722..8fe9d65 100644
@@ -6,7 +6,7 @@
 # it under the terms of the GNU General Public License (version 2 only)
 # as published by the Free Software Foundation.
 
-common-objs = $(addprefix ../../../virt/kvm/, kvm_main.o)
+common-objs = $(addprefix ../../../virt/kvm/, kvm_main.o eventfd.o)
 
 ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 
arch/s390/kvm/diag.c +0 -0
index a390687..13cf74a 100644
@@ -104,6 +104,29 @@ static int __diag_ipl_functions(struct kvm_vcpu *vcpu)
 	return -EREMOTE;

arch/s390/kvm/diag.c static int __diag_ipl_functions(struct kvm_vcpu *vcpu) · line 114

Quoted 10 lines
arch/s390/kvm/diag.c +8 -0
 }
 
+static int __diag_virtio_hypercall(struct kvm_vcpu *vcpu)
+{
+	int ret, idx;
+
+	/* No virtio-ccw notification? Get out quickly. */
+	if (!vcpu->kvm->arch.css_support ||
+	    (vcpu->run->s.regs.gprs[1] != 3))
+		return -EOPNOTSUPP;

Looks like we have:
arch/s390/include/uapi/asm/kvm_virtio.h:#define KVM_S390_VIRTIO_NOTIFY 0 arch/s390/include/uapi/asm/kvm_virtio.h:#define KVM_S390_VIRTIO_RESET 1
arch/s390/include/uapi/asm/kvm_virtio.h:#define KVM_S390_VIRTIO_SET_STATUS 2

KVM_S390_VIRTIO_CSS_NOTIFY is missing? Strange. Maybe it should be added in that header and included here? Do some guests use it?
Also, do you want to support KVM_S390_VIRTIO_NOTIFY as well?

Quoted 41 lines
arch/s390/kvm/diag.c +17 -0
+
+	idx = srcu_read_lock(&vcpu->kvm->srcu);
+	/*
+	 * The layout is as follows:
+	 * - gpr 2 contains the subchannel id (passed as addr)
+	 * - gpr 3 contains the virtqueue index (passed as datamatch)
+	 */
+	ret = kvm_io_bus_write(vcpu->kvm, KVM_CSS_BUS,
+				vcpu->run->s.regs.gprs[2],
+				8, &vcpu->run->s.regs.gprs[3]);
+	srcu_read_unlock(&vcpu->kvm->srcu, idx);
+	/* kvm_io_bus_write returns -EOPNOTSUPP if it found no match. */
+	return ret;
+}
+
 int kvm_s390_handle_diag(struct kvm_vcpu *vcpu)
 {
 	int code = (vcpu->arch.sie_block->ipb & 0xfff0000) >> 16;
@@ -118,6 +141,8 @@ int kvm_s390_handle_diag(struct kvm_vcpu *vcpu)
 		return __diag_time_slice_end_directed(vcpu);
 	case 0x308:
 		return __diag_ipl_functions(vcpu);
+	case 0x500:
+		return __diag_virtio_hypercall(vcpu);
 	default:
 		return -EOPNOTSUPP;
 	}
arch/s390/kvm/kvm-s390.c +1 -0
index f822d36..04d2454 100644
@@ -142,6 +142,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_ONE_REG:
 	case KVM_CAP_ENABLE_CAP:
 	case KVM_CAP_S390_CSS_SUPPORT:
+	case KVM_CAP_IOEVENTFD:
 		r = 1;
 		break;
 	case KVM_CAP_NR_VCPUS:
Signature · 1 line
1.7.12.4
#10 lore raw

Re: [RFC PATCH v2 4/4] KVM: s390: Wire up ioeventfd.

Quoted 59 lines · On Sun, 24 Feb 2013 11:45:22 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 42 lines · Cornelia Huck

Enable ioeventfd support on s390 and hook up diagnose 500 virtio-ccw notifications.

---
 arch/s390/kvm/Kconfig    |  1 +
 arch/s390/kvm/Makefile   |  2 +-
 arch/s390/kvm/diag.c     | 25 +++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c |  1 +
 4 files changed, 28 insertions(+), 1 deletion(-)
arch/s390/kvm/Kconfig +1 -0
index b58dd86..3c43e30 100644
@@ -22,6 +22,7 @@ config KVM
 	select PREEMPT_NOTIFIERS
 	select ANON_INODES
 	select HAVE_KVM_CPU_RELAX_INTERCEPT
+	select HAVE_KVM_EVENTFD
 	---help---
 	  Support hosting paravirtualized guest machines using the SIE
 	  virtualization capability on the mainframe. This should work
arch/s390/kvm/Makefile +1 -1
index 3975722..8fe9d65 100644
@@ -6,7 +6,7 @@
 # it under the terms of the GNU General Public License (version 2 only)
 # as published by the Free Software Foundation.
 
-common-objs = $(addprefix ../../../virt/kvm/, kvm_main.o)
+common-objs = $(addprefix ../../../virt/kvm/, kvm_main.o eventfd.o)
 
 ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 
arch/s390/kvm/diag.c +0 -0
index a390687..13cf74a 100644
@@ -104,6 +104,29 @@ static int __diag_ipl_functions(struct kvm_vcpu *vcpu)
 	return -EREMOTE;

arch/s390/kvm/diag.c static int __diag_ipl_functions(struct kvm_vcpu *vcpu) · line 114

Quoted 10 lines
arch/s390/kvm/diag.c +8 -0
 }
 
+static int __diag_virtio_hypercall(struct kvm_vcpu *vcpu)
+{
+	int ret, idx;
+
+	/* No virtio-ccw notification? Get out quickly. */
+	if (!vcpu->kvm->arch.css_support ||
+	    (vcpu->run->s.regs.gprs[1] != 3))
+		return -EOPNOTSUPP;

Looks like we have:
arch/s390/include/uapi/asm/kvm_virtio.h:#define KVM_S390_VIRTIO_NOTIFY 0 arch/s390/include/uapi/asm/kvm_virtio.h:#define KVM_S390_VIRTIO_RESET 1
arch/s390/include/uapi/asm/kvm_virtio.h:#define KVM_S390_VIRTIO_SET_STATUS 2

Quoted 2 lines

KVM_S390_VIRTIO_CSS_NOTIFY is missing? Strange. Maybe it should be added in that header and included here?

kvm_virtio.h is for the old-style s390-virtio mechanism. Subcode 3 will be included in 3.9.

Quoted 2 lines

Do some guests use it?
Also, do you want to support KVM_S390_VIRTIO_NOTIFY as well?

We don't want to invest any more effort in the old style s390-virtio transport (other than keeping it working). Any future work will be for virtio-ccw.

Quoted 44 lines
Quoted 43 lines
arch/s390/kvm/diag.c +17 -0
+
+	idx = srcu_read_lock(&vcpu->kvm->srcu);
+	/*
+	 * The layout is as follows:
+	 * - gpr 2 contains the subchannel id (passed as addr)
+	 * - gpr 3 contains the virtqueue index (passed as datamatch)
+	 */
+	ret = kvm_io_bus_write(vcpu->kvm, KVM_CSS_BUS,
+				vcpu->run->s.regs.gprs[2],
+				8, &vcpu->run->s.regs.gprs[3]);
+	srcu_read_unlock(&vcpu->kvm->srcu, idx);
+	/* kvm_io_bus_write returns -EOPNOTSUPP if it found no match. */
+	return ret;
+}
+
 int kvm_s390_handle_diag(struct kvm_vcpu *vcpu)
 {
 	int code = (vcpu->arch.sie_block->ipb & 0xfff0000) >> 16;
@@ -118,6 +141,8 @@ int kvm_s390_handle_diag(struct kvm_vcpu *vcpu)
 		return __diag_time_slice_end_directed(vcpu);
 	case 0x308:
 		return __diag_ipl_functions(vcpu);
+	case 0x500:
+		return __diag_virtio_hypercall(vcpu);
 	default:
 		return -EOPNOTSUPP;
 	}
arch/s390/kvm/kvm-s390.c +1 -0
index f822d36..04d2454 100644
@@ -142,6 +142,7 @@ int kvm_dev_ioctl_check_extension(long ext)
 	case KVM_CAP_ONE_REG:
 	case KVM_CAP_ENABLE_CAP:
 	case KVM_CAP_S390_CSS_SUPPORT:
+	case KVM_CAP_IOEVENTFD:
 		r = 1;
 		break;
 	case KVM_CAP_NR_VCPUS:
Signature · 1 line
1.7.12.4
#11 lore raw
In reply to Cornelia Huck
Quoted 8 lines · Cornelia Huck

Here's the second attempt at implementing ioeventfd for s390.

Rather than the architecture-specific functions used in v1, we now try to integrate with the kvm_io_device infrastructure. Calls to diagnose 500 subcode 3 are now mapped to _write. These devices are created on a new KVM_CSS_BUS when using a new flag KVM_IOEVENTFD_FLAG_CSS. addr and datamatch are (ab)used to contain the subchannel id and the virtqueue.

OK so a question: qemu has:

	#define KVM_S390_VIRTIO_NOTIFY          0
	#define KVM_S390_VIRTIO_RESET           1
	#define KVM_S390_VIRTIO_SET_STATUS      2
	#define KVM_S390_VIRTIO_CCW_NOTIFY      3

So this is really s390_virtio_ccw not generally css? Maybe should say so in all headers.

Quoted 31 lines

we want to attach other hypercalls or carry more payload.

Another limitation is the limit of 1000 io devices per bus, which we would hit easily with a few hundred devices, but that should be fixable.

v1 -> v2:
- Move irqfd initialization from a module init function to kvm_init, eliminating the need for a second module for kvm/s390.
- Use kvm_io_device for s390 css devices.

Cornelia Huck (4):
KVM: Initialize irqfd from kvm_init().
KVM: Introduce KVM_CSS_BUS.
KVM: ioeventfd for s390 css devices.
KVM: s390: Wire up ioeventfd.

 Documentation/virtual/kvm/api.txt |  7 +++++++
 arch/s390/kvm/Kconfig             |  1 +
 arch/s390/kvm/Makefile            |  2 +-
 arch/s390/kvm/diag.c              | 25 +++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c          |  1 +
 include/linux/kvm_host.h          | 14 ++++++++++++++
 include/uapi/linux/kvm.h          |  2 ++
 virt/kvm/eventfd.c                | 15 ++++++++-------
 virt/kvm/kvm_main.c               |  6 ++++++
 9 files changed, 65 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.12.4
#12 lore raw
In reply to Cornelia Huck
Quoted 1 line · Cornelia Huck

Here's the second attempt at implementing ioeventfd for s390.

The patchset looks fine overall.
Minor comments and questions below.

Quoted 10 lines

Rather than the architecture-specific functions used in v1, we now try to integrate with the kvm_io_device infrastructure. Calls to diagnose 500 subcode 3 are now mapped to _write. These devices are created on a new KVM_CSS_BUS when using a new flag KVM_IOEVENTFD_FLAG_CSS. addr and datamatch are (ab)used to contain the subchannel id and the virtqueue.

A drawback is that this interface is not easily extendable should we want to attach other hypercalls or carry more payload.

Under s390 kvm_hypercallX already uses diagnose 500 so that seems fine. If you want to make it more generic and support more subcodes, I think you'll have to pass an extra u64 field:
to bus to both avoid overflowing int value and avoid ugly bus-specific hacks in generic code.

Will we ever need that? Code using subcode 3 does not yet seem to be upstream in 3.8 so maybe yes, but you decide. An alternative is to add new bus types when kvm needs to handle new subcodes. So e.g. KVM_BUS_S390_VIRTIO_CCW_NOTIFY and KVM_BUS_S390_VIRTIO_NOTIFY ?

You decide, I'm fine with either approach.

More minor comments and questions in response to individual patches.

Quoted 29 lines

Another limitation is the limit of 1000 io devices per bus, which we would hit easily with a few hundred devices, but that should be fixable.

v1 -> v2:
- Move irqfd initialization from a module init function to kvm_init, eliminating the need for a second module for kvm/s390.
- Use kvm_io_device for s390 css devices.

Cornelia Huck (4):
KVM: Initialize irqfd from kvm_init().
KVM: Introduce KVM_CSS_BUS.
KVM: ioeventfd for s390 css devices.
KVM: s390: Wire up ioeventfd.

 Documentation/virtual/kvm/api.txt |  7 +++++++
 arch/s390/kvm/Kconfig             |  1 +
 arch/s390/kvm/Makefile            |  2 +-
 arch/s390/kvm/diag.c              | 25 +++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c          |  1 +
 include/linux/kvm_host.h          | 14 ++++++++++++++
 include/uapi/linux/kvm.h          |  2 ++
 virt/kvm/eventfd.c                | 15 ++++++++-------
 virt/kvm/kvm_main.c               |  6 ++++++
 9 files changed, 65 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.12.4
#13 lore raw
Quoted 5 lines · On Sun, 24 Feb 2013 11:56:39 +0200 "Michael S. Tsirkin" <[email protected]> wrote:
Quoted 1 line · Cornelia Huck

Here's the second attempt at implementing ioeventfd for s390.

The patchset looks fine overall.
Minor comments and questions below.

Cool, thanks for reviewing.

Quoted 12 lines
Quoted 11 lines

Rather than the architecture-specific functions used in v1, we now try to integrate with the kvm_io_device infrastructure. Calls to diagnose 500 subcode 3 are now mapped to _write. These devices are created on a new KVM_CSS_BUS when using a new flag KVM_IOEVENTFD_FLAG_CSS. addr and datamatch are (ab)used to contain the subchannel id and the virtqueue.

A drawback is that this interface is not easily extendable should we want to attach other hypercalls or carry more payload.

Quoted 8 lines

Under s390 kvm_hypercallX already uses diagnose 500 so that seems fine. If you want to make it more generic and support more subcodes, I think you'll have to pass an extra u64 field:
to bus to both avoid overflowing int value and avoid ugly bus-specific hacks in generic code.

Will we ever need that? Code using subcode 3 does not yet seem to be upstream in 3.8 so maybe yes, but you decide.

Subcode 3 will be in 3.9.

Quoted 3 lines

An alternative is to add new bus types when kvm needs to handle new subcodes. So e.g. KVM_BUS_S390_VIRTIO_CCW_NOTIFY and KVM_BUS_S390_VIRTIO_NOTIFY ?

I think I'll fall back to a new bus type should we ever need a new notification type - the less strange hacks, the better.

Quoted 35 lines

You decide, I'm fine with either approach.

More minor comments and questions in response to individual patches.

Quoted 29 lines

Another limitation is the limit of 1000 io devices per bus, which we would hit easily with a few hundred devices, but that should be fixable.

v1 -> v2:
- Move irqfd initialization from a module init function to kvm_init, eliminating the need for a second module for kvm/s390.
- Use kvm_io_device for s390 css devices.

Cornelia Huck (4):
KVM: Initialize irqfd from kvm_init().
KVM: Introduce KVM_CSS_BUS.
KVM: ioeventfd for s390 css devices.
KVM: s390: Wire up ioeventfd.

 Documentation/virtual/kvm/api.txt |  7 +++++++
 arch/s390/kvm/Kconfig             |  1 +
 arch/s390/kvm/Makefile            |  2 +-
 arch/s390/kvm/diag.c              | 25 +++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.c          |  1 +
 include/linux/kvm_host.h          | 14 ++++++++++++++
 include/uapi/linux/kvm.h          |  2 ++
 virt/kvm/eventfd.c                | 15 ++++++++-------
 virt/kvm/kvm_main.c               |  6 ++++++
 9 files changed, 65 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.12.4