Skip to contentSkip to conversation

KVM: Improve MMIO Coalescing API

PATCH In review Ilias Stamatis 3 participants 13 messages Aug 20, 2024 – Aug 29, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v3 cover KVM: Improve MMIO Coalescing API

The current MMIO coalescing design has a few drawbacks which limit its usefulness. Currently all coalesced MMIO zones use the same ring buffer. That means that upon a userspace exit we have to handle potentially unrelated MMIO writes synchronously. And a VM-wide lock needs to be taken in the kernel when an MMIO exit occurs.

Additionally, there is no direct way for userspace to be notified about coalesced MMIO writes. If the next MMIO exit to userspace is when the ring buffer has filled then a substantial (and unbounded) amount of time may have passed since the first coalesced MMIO.

This series adds new ioctls to KVM that allow for greater control by making it possible to associate different MMIO zones with different ring buffers. It also allows userspace to use poll() to check for coalesced writes in order to avoid userspace exits in vCPU threads (see patch 3 for why this can be useful).

The idea of improving the API in this way originally came from Paul Durrant ([email protected]) but the implementation is mine.

The first patch in the series is a bug in the existing code that I discovered while writing a selftest and can be merged independently.

Changelog:

v3:
- Fixed test robot build warning and made small changes suggested by Sean Christopherson

v2: https://lore.kernel.org/kvm/[email protected]/T/

v1: https://lore.kernel.org/kvm/[email protected]/T/

Ilias Stamatis (6):
KVM: Fix coalesced_mmio_has_room()
KVM: Add KVM_CREATE_COALESCED_MMIO_BUFFER ioctl
KVM: Support poll() on coalesced mmio buffer fds
KVM: Add KVM_(UN)REGISTER_COALESCED_MMIO2 ioctls
KVM: Documentation: Document v2 of coalesced MMIO API
KVM: selftests: Add coalesced_mmio_test

 Documentation/virt/kvm/api.rst                |  91 +++++
 include/linux/kvm_host.h                      |   1 +
 include/uapi/linux/kvm.h                      |  18 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/coalesced_mmio_test.c       | 313 ++++++++++++++++++
 virt/kvm/coalesced_mmio.c                     | 203 +++++++++++-
 virt/kvm/coalesced_mmio.h                     |  17 +-
 virt/kvm/kvm_main.c                           |  37 ++-
 8 files changed, 661 insertions(+), 20 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/coalesced_mmio_test.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH v3 1/6 KVM: Fix coalesced_mmio_has_room()

The following calculation used in coalesced_mmio_has_room() to check whether the ring buffer is full is wrong and only allows half the buffer to be used.

avail = (ring->first - last - 1) % KVM_COALESCED_MMIO_MAX; if (avail == 0)

	/* full */

The % operator in C is not the modulo operator but the remainder operator. Modulo and remainder operators differ with respect to negative values. But all values are unsigned in this case anyway.

The above might have worked as expected in python for example:

Quoted 1 line
Quoted 1 line
Quoted 1 line

(-86) % 170

84

However it doesn't work the same way in C.

printf("avail: %d\n", (-86) % 170);
printf("avail: %u\n", (-86) % 170);
printf("avail: %u\n", (-86u) % 170u);

Using gcc-11 these print:

avail: -86
avail: 4294967210
avail: 0

Fix the calculation and allow all but one entries in the buffer to be used as originally intended.

Fixes 105f8d40a737 ("KVM: Calculate available entries in coalesced mmio ring")
---
 virt/kvm/coalesced_mmio.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
virt/kvm/coalesced_mmio.c +1 -3
index 1b90acb6e3fe..184c5c40c9c1 100644
@@ -43,7 +43,6 @@ static int coalesced_mmio_in_range(struct kvm_coalesced_mmio_dev *dev,
 static int coalesced_mmio_has_room(struct kvm_coalesced_mmio_dev *dev, u32 last)
 {
 	struct kvm_coalesced_mmio_ring *ring;
-	unsigned avail;
 
 	/* Are we able to batch it ? */
 
@@ -52,8 +51,7 @@ static int coalesced_mmio_has_room(struct kvm_coalesced_mmio_dev *dev, u32 last)
 	 * there is always one unused entry in the buffer
 	 */
 	ring = dev->kvm->coalesced_mmio_ring;
-	avail = (ring->first - last - 1) % KVM_COALESCED_MMIO_MAX;
-	if (avail == 0) {
+	if ((last + 1) % KVM_COALESCED_MMIO_MAX == READ_ONCE(ring->first)) {
 		/* full */
 		return 0;
 	}
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH v3 1/6] KVM: Fix coalesced_mmio_has_room()

Quoted 28 lines · Ilias Stamatis

The following calculation used in coalesced_mmio_has_room() to check whether the ring buffer is full is wrong and only allows half the buffer to be used.

avail = (ring->first - last - 1) % KVM_COALESCED_MMIO_MAX; if (avail == 0)

	/* full */

The % operator in C is not the modulo operator but the remainder operator. Modulo and remainder operators differ with respect to negative values. But all values are unsigned in this case anyway.

The above might have worked as expected in python for example:

Quoted 1 line
Quoted 1 line
Quoted 1 line

(-86) % 170

84

However it doesn't work the same way in C.

printf("avail: %d\n", (-86) % 170);
printf("avail: %u\n", (-86) % 170);
printf("avail: %u\n", (-86u) % 170u);

Using gcc-11 these print:

avail: -86
avail: 4294967210
avail: 0

Quoted 7 lines

Fix the calculation and allow all but one entries in the buffer to be used as originally intended.

Fixes 105f8d40a737 ("KVM: Calculate available entries in coalesced mmio ring")

---

Doh, I applied v2 instead of v3. Though unless mine eyes deceive me, they're the same.

#3 lore raw

PATCH v3 2/6 KVM: Add KVM_CREATE_COALESCED_MMIO_BUFFER ioctl

In reply to Ilias Stamatis

The current MMIO coalescing design has a few drawbacks which limit its usefulness. Currently all coalesced MMIO zones use the same ring buffer. That means that upon a userspace exit we have to handle potentially unrelated MMIO writes synchronously. And a VM-wide lock needs to be taken in the kernel when an MMIO exit occurs.

Additionally, there is no direct way for userspace to be notified about coalesced MMIO writes. If the next MMIO exit to userspace is when the ring buffer has filled then a substantial (and unbounded) amount of time may have passed since the first coalesced MMIO.

Add a KVM_CREATE_COALESCED_MMIO_BUFFER ioctl to KVM. This ioctl simply returns a file descriptor to the caller but does not allocate a ring buffer. Userspace can then pass this fd to mmap() to actually allocate a buffer and map it to its address space.

Subsequent patches will allow userspace to:

- Associate the fd with a coalescing zone when registering it so that writes to that zone are accumulated in that specific ring buffer rather than the VM-wide one.
- Poll for MMIO writes using this fd.

---

v2->v3:
- Removed unnecessary brackets in a switch case and adjusted indentation on a spinlock as suggested by Sean Christopherson

 include/linux/kvm_host.h  |   1 +
 include/uapi/linux/kvm.h  |   2 +
 virt/kvm/coalesced_mmio.c | 141 +++++++++++++++++++++++++++++++++++---
 virt/kvm/coalesced_mmio.h |   9 +++
 virt/kvm/kvm_main.c       |   3 +
 5 files changed, 148 insertions(+), 8 deletions(-)
include/linux/kvm_host.h +1 -0
index ed0520268de4..efb07422e76b 100644
@@ -808,6 +808,7 @@ struct kvm {
 	struct kvm_coalesced_mmio_ring *coalesced_mmio_ring;
 	spinlock_t ring_lock;
 	struct list_head coalesced_zones;
+	struct list_head coalesced_buffers;
 #endif
 
 	struct mutex irq_lock;
include/uapi/linux/kvm.h +2 -0
index 637efc055145..87f79a820fc0 100644
@@ -1573,4 +1573,6 @@ struct kvm_pre_fault_memory {
 	__u64 padding[5];
 };
 
+#define KVM_CREATE_COALESCED_MMIO_BUFFER _IO(KVMIO,   0xd6)
+
 #endif /* __LINUX_KVM_H */
virt/kvm/coalesced_mmio.c +133 -8

209 lines not on this page · raw message

virt/kvm/coalesced_mmio.h +9 -0
index 36f84264ed25..37d9d8f325bb 100644
@@ -20,6 +20,14 @@ struct kvm_coalesced_mmio_dev {
 	struct kvm_io_device dev;
 	struct kvm *kvm;
 	struct kvm_coalesced_mmio_zone zone;
+	struct kvm_coalesced_mmio_buffer_dev *buffer_dev;
+};
+
+struct kvm_coalesced_mmio_buffer_dev {
+	struct list_head list;
+	struct kvm *kvm;
+	spinlock_t ring_lock;
+	struct kvm_coalesced_mmio_ring *ring;
 };
 
 int kvm_coalesced_mmio_init(struct kvm *kvm);
@@ -28,6 +36,7 @@ int kvm_vm_ioctl_register_coalesced_mmio(struct kvm *kvm,
 					struct kvm_coalesced_mmio_zone *zone);
 int kvm_vm_ioctl_unregister_coalesced_mmio(struct kvm *kvm,
 					struct kvm_coalesced_mmio_zone *zone);
+int kvm_vm_ioctl_create_coalesced_mmio_buffer(struct kvm *kvm);
 
 #else
 
virt/kvm/kvm_main.c +3 -0
index 238940c3cb32..9f6ad6e03317 100644
@@ -5244,6 +5244,9 @@ static long kvm_vm_ioctl(struct file *filp,
 		r = kvm_vm_ioctl_unregister_coalesced_mmio(kvm, &zone);
 		break;
 	}
+	case KVM_CREATE_COALESCED_MMIO_BUFFER:
+		r = kvm_vm_ioctl_create_coalesced_mmio_buffer(kvm);
+		break;
 #endif
 	case KVM_IRQFD: {
 		struct kvm_irqfd data;
Signature · 1 line
2.34.1
#4 lore raw

Re: [PATCH v3 2/6] KVM: Add KVM_CREATE_COALESCED_MMIO_BUFFER ioctl

Quoted 6 lines · Ilias Stamatis

The current MMIO coalescing design has a few drawbacks which limit its usefulness. Currently all coalesced MMIO zones use the same ring buffer. That means that upon a userspace exit we have to handle potentially unrelated MMIO writes synchronously. And a VM-wide lock needs to be taken in the kernel when an MMIO exit occurs.

Quoted 6 lines

Additionally, there is no direct way for userspace to be notified about coalesced MMIO writes. If the next MMIO exit to userspace is when the ring buffer has filled then a substantial (and unbounded) amount of time may have passed since the first coalesced MMIO.

Add a KVM_CREATE_COALESCED_MMIO_BUFFER ioctl to KVM. This ioctl simply

Why allocate the buffer in KVM? It allows reusing coalesced_mmio_write() without needing {get,put}_user() or any form of kmap(), but it adds complexity (quite a bit in fact) to KVM and inherits many of the restrictions and warts of the existing coalesced I/O implementation.

E.g. the size of the ring buffer is "fixed", yet variable based on the host kernel page size.

Why not go with a BYOB model? E.g. so that userspace can explicitly define the buffer size to best fit the properties of the I/O range, and to avoid additional complexity in KVM.

Quoted 10 lines

returns a file descriptor to the caller but does not allocate a ring buffer. Userspace can then pass this fd to mmap() to actually allocate a buffer and map it to its address space.

Subsequent patches will allow userspace to:

- Associate the fd with a coalescing zone when registering it so that writes to that zone are accumulated in that specific ring buffer rather than the VM-wide one.
- Poll for MMIO writes using this fd.

Why? I get the desire for a doorbell, but KVM already supports "fast" I/O for doorbells. The only use case I can think of is where the doorbell sits in the same region as the data payload, but that essentially just saves an entry in KVM's MMIO bus (or I suppose two entries if the doorbell is in the middle of the region).

Quoted 4 lines

---

...

virt/kvm/coalesced_mmio.c void kvm_coalesced_mmio_free(struct kvm *kvm) · line 154

Quoted 10 lines
virt/kvm/coalesced_mmio.c +10 -0
+static void coalesced_mmio_buffer_vma_close(struct vm_area_struct *vma)
+{
+	struct kvm_coalesced_mmio_buffer_dev *dev = vma->vm_private_data;
+
+	spin_lock(&dev->ring_lock);
+
+	vfree(dev->ring);
+	dev->ring = NULL;
+
+	spin_unlock(&dev->ring_lock);

I doubt this is correct. I don't see VM_DONTCOPY, and so userspace can create a second (or more) VMA by forking, and then KVM will hit a UAF if any of the VMAs is closed.

Quoted 5 lines
virt/kvm/coalesced_mmio.c +5 -0
+}
+
+static const struct vm_operations_struct coalesced_mmio_buffer_vm_ops = {
+	.close = coalesced_mmio_buffer_vma_close,
+};

virt/kvm/coalesced_mmio.c void kvm_coalesced_mmio_free(struct kvm *kvm) · line 169

Quoted 10 lines
virt/kvm/coalesced_mmio.c +10 -0
+
+static int coalesced_mmio_buffer_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	struct kvm_coalesced_mmio_buffer_dev *dev = file->private_data;
+	unsigned long pfn;
+	int ret = 0;
+
+	spin_lock(&dev->ring_lock);
+
+	if (dev->ring) {

Restricting userspace to a single mmap() is a very bizarre restriction.

Quoted 3 lines
virt/kvm/coalesced_mmio.c +3 -0
+		ret = -EBUSY;
+		goto out_unlock;
+	}
#5 lore raw

Re: [PATCH v3 2/6] KVM: Add KVM_CREATE_COALESCED_MMIO_BUFFER ioctl

Quoted 19 lines · Sean Christopherson
Quoted 6 lines · Ilias Stamatis

The current MMIO coalescing design has a few drawbacks which limit its usefulness. Currently all coalesced MMIO zones use the same ring buffer. That means that upon a userspace exit we have to handle potentially unrelated MMIO writes synchronously. And a VM-wide lock needs to be taken in the kernel when an MMIO exit occurs.

Quoted 6 lines

Additionally, there is no direct way for userspace to be notified about coalesced MMIO writes. If the next MMIO exit to userspace is when the ring buffer has filled then a substantial (and unbounded) amount of time may have passed since the first coalesced MMIO.

Add a KVM_CREATE_COALESCED_MMIO_BUFFER ioctl to KVM. This ioctl simply

Why allocate the buffer in KVM? It allows reusing coalesced_mmio_write() without needing {get,put}_user() or any form of kmap(), but it adds complexity (quite a bit in fact) to KVM and inherits many of the restrictions and warts of the existing coalesced I/O implementation.

Quoted 6 lines

E.g. the size of the ring buffer is "fixed", yet variable based on the host kernel page size.

Why not go with a BYOB model? E.g. so that userspace can explicitly define the buffer size to best fit the properties of the I/O range, and to avoid additional complexity in KVM.

Fair enough. I used the original implementation as a guide, but let's do it Right™ this time.

Quoted 7 lines
Quoted 7 lines

returns a file descriptor to the caller but does not allocate a ring buffer. Userspace can then pass this fd to mmap() to actually allocate a buffer and map it to its address space.

Subsequent patches will allow userspace to:

Quoted 7 lines
Quoted 4 lines

- Associate the fd with a coalescing zone when registering it so that writes to that zone are accumulated in that specific ring buffer rather than the VM-wide one.
- Poll for MMIO writes using this fd.

Why? I get the desire for a doorbell, but KVM already supports "fast" I/O for doorbells.

What do you refer to here? ioeventfd?

Correct me if I am wrong, but my understanding is that with an ioeventfd the write value is not available. And that is a problem when that value is a head or tail pointer.

I also realised that I need to improve my commit messages.

Quoted 13 lines

The only use case I can think of is where the doorbell sits in the
same region as the data payload, but that essentially just saves an entry in KVM's MMIO bus (or I suppose two entries if the doorbell is in the middle of the region).

Quoted 4 lines

---

...

Quoted 14 lines

virt/kvm/coalesced_mmio.c void kvm_coalesced_mmio_free(struct kvm *kvm) · line 154

Quoted 10 lines
virt/kvm/coalesced_mmio.c +10 -0
+static void coalesced_mmio_buffer_vma_close(struct vm_area_struct *vma)
+{
+	struct kvm_coalesced_mmio_buffer_dev *dev = vma->vm_private_data;
+
+	spin_lock(&dev->ring_lock);
+
+	vfree(dev->ring);
+	dev->ring = NULL;
+
+	spin_unlock(&dev->ring_lock);

I doubt this is correct. I don't see VM_DONTCOPY, and so userspace can create a second (or more) VMA by forking, and then KVM will hit a UAF if any of the VMAs is closed.

Oops. Will fix.

Quoted 10 lines
Quoted 10 lines
virt/kvm/coalesced_mmio.c +9 -0
+}
+
+static const struct vm_operations_struct coalesced_mmio_buffer_vm_ops = {
+	.close = coalesced_mmio_buffer_vma_close,
+};
+
+static int coalesced_mmio_buffer_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	struct kvm_coalesced_mmio_buffer_dev *dev = file->private_data;
Quoted 9 lines

virt/kvm/coalesced_mmio.c void kvm_coalesced_mmio_free(struct kvm *kvm) · line 169

Quoted 6 lines
virt/kvm/coalesced_mmio.c +6 -0
+	unsigned long pfn;
+	int ret = 0;
+
+	spin_lock(&dev->ring_lock);
+
+	if (dev->ring) {

Restricting userspace to a single mmap() is a very bizarre restriction.

Hmm, you are right. Even if the buffer is allocated in the kernel, and one was allocated already we should let the user map it to multiple addresses. I will fix that too.

Quoted 3 lines

virt/kvm/coalesced_mmio.c void kvm_coalesced_mmio_free(struct kvm *kvm) · line 172

Quoted 3 lines
virt/kvm/coalesced_mmio.c +3 -0
+		ret = -EBUSY;
+		goto out_unlock;
+	}

Thanks for the review.

Ilias

#6 lore raw

Re: [PATCH v3 2/6] KVM: Add KVM_CREATE_COALESCED_MMIO_BUFFER ioctl

Quoted 12 lines · Ilias Stamatis
Quoted 11 lines · Sean Christopherson
Quoted 10 lines

returns a file descriptor to the caller but does not allocate a ring buffer. Userspace can then pass this fd to mmap() to actually allocate a buffer and map it to its address space.

Subsequent patches will allow userspace to:

- Associate the fd with a coalescing zone when registering it so that writes to that zone are accumulated in that specific ring buffer rather than the VM-wide one.
- Poll for MMIO writes using this fd.

Quoted 4 lines
Quoted 2 lines

Why? I get the desire for a doorbell, but KVM already supports "fast" I/O for doorbells.

What do you refer to here? ioeventfd?

Ya.

Quoted 3 lines

Correct me if I am wrong, but my understanding is that with an ioeventfd the write value is not available. And that is a problem when that value is a head or tail pointer.

Ah. Can you describe (or point at) an example device? I don't read many device specs (understatement). It would be helpful to have a concrete use case for reviewing the design itself.

In a perfect world, poll() support would come from a third party file type, as this doesn't seem _that_ unique (to KVM). But AFAICT there isn't an existing type that is a good fit, probably because it's such a simple thing.

#7 lore raw

Re: [PATCH v3 2/6] KVM: Add KVM_CREATE_COALESCED_MMIO_BUFFER ioctl

Quoted 20 lines · Sean Christopherson
Quoted 12 lines · Ilias Stamatis
Quoted 11 lines · Sean Christopherson
Quoted 10 lines

returns a file descriptor to the caller but does not allocate a ring buffer. Userspace can then pass this fd to mmap() to actually allocate a buffer and map it to its address space.

Subsequent patches will allow userspace to:

- Associate the fd with a coalescing zone when registering it so that writes to that zone are accumulated in that specific ring buffer rather than the VM-wide one.
- Poll for MMIO writes using this fd.

Quoted 4 lines
Quoted 2 lines

Why? I get the desire for a doorbell, but KVM already supports "fast" I/O for doorbells.

What do you refer to here? ioeventfd?

Ya.

Quoted 7 lines
Quoted 3 lines

Correct me if I am wrong, but my understanding is that with an ioeventfd the write value is not available. And that is a problem when that value is a head or tail pointer.

Ah. Can you describe (or point at) an example device? I don't read many device specs (understatement). It would be helpful to have a concrete use case for reviewing the design itself.

Intel 82599 VF is one such example. I believe NVMe is another example.

Quoted 4 lines

In a perfect world, poll() support would come from a third party file type, as this doesn't seem _that_ unique (to KVM). But AFAICT there isn't an existing type that is a good fit, probably because it's such a simple thing.

#8 lore raw

PATCH v3 3/6 KVM: Support poll() on coalesced mmio buffer fds

In reply to Ilias Stamatis

There is no direct way for userspace to be notified about coalesced MMIO writes when using KVM_REGISTER_COALESCED_MMIO. If the next MMIO exit is when the ring buffer has filled then a substantial (and unbounded) amount of time may have passed since the first coalesced MMIO.

To improve this, make it possible for userspace to use poll() and select() on the fd returned by the KVM_CREATE_COALESCED_MMIO_BUFFER ioctl. This way a userspace VMM could have dedicated threads that deal with writes to specific MMIO zones.

For example, a common use of MMIO, particularly in the realm of network devices, is as a doorbell. A write to a doorbell register will trigger the device to initiate a DMA transfer.

When a network device is emulated by userspace a write to a doorbell register would typically result in an MMIO exit so that userspace can emulate the DMA transfer in a timely manner. No further processing can be done until userspace performs the necessary emulation and re-invokes KVM_RUN. Even if userspace makes use of another thread to emulate the DMA transfer such MMIO exits are disruptive to the vCPU and they may also be quite frequent if, for example, the vCPU is sending a sequence of short packets to the network device.

By supporting poll() on coalesced buffer fds, userspace can have dedicated threads wait for new doorbell writes and avoid the performance hit of userspace exits on the main vCPU threads.

---

v2->v3:
- Changed POLLIN | POLLRDNORM to EPOLLIN | EPOLLRDNORM

 virt/kvm/coalesced_mmio.c | 22 ++++++++++++++++++++++
 virt/kvm/coalesced_mmio.h |  1 +
 2 files changed, 23 insertions(+)
virt/kvm/coalesced_mmio.c +22 -0
index 98b7e8760aa7..039c6ffcb2a8 100644
@@ -16,6 +16,7 @@
 #include <linux/slab.h>
 #include <linux/kvm.h>
 #include <linux/anon_inodes.h>
+#include <linux/poll.h>
 
 #include "coalesced_mmio.h"
 
@@ -97,6 +98,10 @@ static int coalesced_mmio_write(struct kvm_vcpu *vcpu,
 	smp_wmb();
 	ring->last = (insert + 1) % KVM_COALESCED_MMIO_MAX;
 	spin_unlock(lock);
+
+	if (dev->buffer_dev)
+		wake_up_interruptible(&dev->buffer_dev->wait_queue);
+
 	return 0;
 }
 
@@ -223,9 +228,25 @@ static int coalesced_mmio_buffer_release(struct inode *inode, struct file *file)
 	return 0;
 }
 
+static __poll_t coalesced_mmio_buffer_poll(struct file *file, struct poll_table_struct *wait)
+{
+	struct kvm_coalesced_mmio_buffer_dev *dev = file->private_data;
+	__poll_t mask = 0;
+
+	poll_wait(file, &dev->wait_queue, wait);
+
+	spin_lock(&dev->ring_lock);
+	if (dev->ring && (READ_ONCE(dev->ring->first) != READ_ONCE(dev->ring->last)))
+		mask = EPOLLIN | EPOLLRDNORM;
+	spin_unlock(&dev->ring_lock);
+
+	return mask;
+}
+
 static const struct file_operations coalesced_mmio_buffer_ops = {
 	.mmap = coalesced_mmio_buffer_mmap,
 	.release = coalesced_mmio_buffer_release,
+	.poll = coalesced_mmio_buffer_poll,
 };
 
 int kvm_vm_ioctl_create_coalesced_mmio_buffer(struct kvm *kvm)
@@ -239,6 +260,7 @@ int kvm_vm_ioctl_create_coalesced_mmio_buffer(struct kvm *kvm)
 		return -ENOMEM;
 
 	dev->kvm = kvm;
+	init_waitqueue_head(&dev->wait_queue);
 	spin_lock_init(&dev->ring_lock);
 
 	ret = anon_inode_getfd("coalesced_mmio_buf", &coalesced_mmio_buffer_ops,
virt/kvm/coalesced_mmio.h +1 -0
index 37d9d8f325bb..d1807ce26464 100644
@@ -26,6 +26,7 @@ struct kvm_coalesced_mmio_dev {
 struct kvm_coalesced_mmio_buffer_dev {
 	struct list_head list;
 	struct kvm *kvm;
+	wait_queue_head_t wait_queue;
 	spinlock_t ring_lock;
 	struct kvm_coalesced_mmio_ring *ring;
 };
Signature · 1 line
2.34.1
#9 lore raw

PATCH v3 4/6 KVM: Add KVM_(UN)REGISTER_COALESCED_MMIO2 ioctls

In reply to Ilias Stamatis

Add 2 new ioctls, KVM_REGISTER_COALESCED_MMIO2 and KVM_UNREGISTER_COALESCED_MMIO2. These do the same thing as their v1 equivalents except an fd returned by KVM_CREATE_COALESCED_MMIO_BUFFER needs to be passed as an argument to them.

The fd representing a ring buffer is associated with an MMIO region registered for coalescing and all writes to that region are accumulated there. This is in contrast to the v1 API where all regions have to share the same buffer. Nevertheless, userspace code can still use the same ring buffer for multiple zones if it wishes to do so.

Userspace can check for the availability of the new API by checking if the KVM_CAP_COALESCED_MMIO2 capability is supported.

---

v2->v3:
- Changed type of buffer_fd from int to __u32
- Removed 0 initialisation of ret in
kvm_vm_ioctl_register_coalesced_mmio()

 include/uapi/linux/kvm.h  | 16 ++++++++++++++++
 virt/kvm/coalesced_mmio.c | 36 +++++++++++++++++++++++++++++++-----
 virt/kvm/coalesced_mmio.h |  7 ++++---
 virt/kvm/kvm_main.c       | 34 +++++++++++++++++++++++++++++++++-
 4 files changed, 84 insertions(+), 9 deletions(-)
include/uapi/linux/kvm.h +16 -0
index 87f79a820fc0..5e9fcc560cc1 100644
@@ -480,6 +480,16 @@ struct kvm_coalesced_mmio_zone {
 	};
 };
 
+struct kvm_coalesced_mmio_zone2 {
+	__u64 addr;
+	__u32 size;
+	union {
+		__u32 pad;
+		__u32 pio;
+	};
+	__u32 buffer_fd;
+};
+
 struct kvm_coalesced_mmio {
 	__u64 phys_addr;
 	__u32 len;
@@ -933,6 +943,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_PRE_FAULT_MEMORY 236
 #define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237
 #define KVM_CAP_X86_GUEST_MODE 238
+#define KVM_CAP_COALESCED_MMIO2 239
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1573,6 +1584,11 @@ struct kvm_pre_fault_memory {
 	__u64 padding[5];
 };
 
+/* Available with KVM_CAP_COALESCED_MMIO2 */
 #define KVM_CREATE_COALESCED_MMIO_BUFFER _IO(KVMIO,   0xd6)
+#define KVM_REGISTER_COALESCED_MMIO2 \
+			_IOW(KVMIO,  0xd7, struct kvm_coalesced_mmio_zone2)
+#define KVM_UNREGISTER_COALESCED_MMIO2 \
+			_IOW(KVMIO,  0xd8, struct kvm_coalesced_mmio_zone2)
 
 #endif /* __LINUX_KVM_H */
virt/kvm/coalesced_mmio.c +31 -5
index 039c6ffcb2a8..4e237ee66711 100644
@@ -17,6 +17,7 @@
 #include <linux/kvm.h>
 #include <linux/anon_inodes.h>
 #include <linux/poll.h>
+#include <linux/file.h>
 
 #include "coalesced_mmio.h"
 
@@ -278,19 +279,40 @@ int kvm_vm_ioctl_create_coalesced_mmio_buffer(struct kvm *kvm)
 }
 
 int kvm_vm_ioctl_register_coalesced_mmio(struct kvm *kvm,
-					 struct kvm_coalesced_mmio_zone *zone)
+					 struct kvm_coalesced_mmio_zone2 *zone,
+					 bool use_buffer_fd)
 {
 	int ret;
+	struct file *file;
 	struct kvm_coalesced_mmio_dev *dev;
 	struct kvm_coalesced_mmio_buffer_dev *buffer_dev = NULL;
 
 	if (zone->pio != 1 && zone->pio != 0)
 		return -EINVAL;
 
+	if (use_buffer_fd) {
+		file = fget(zone->buffer_fd);
+		if (!file)
+			return -EBADF;
+
+		if (file->f_op != &coalesced_mmio_buffer_ops) {
+			fput(file);
+			return -EINVAL;
+		}
+
+		buffer_dev = file->private_data;
+		if (!buffer_dev->ring) {
+			fput(file);
+			return -ENOBUFS;
+		}
+	}
+
 	dev = kzalloc(sizeof(struct kvm_coalesced_mmio_dev),
 		      GFP_KERNEL_ACCOUNT);
-	if (!dev)
-		return -ENOMEM;
+	if (!dev) {
+		ret = -ENOMEM;
+		goto out_free_file;
+	}
 
 	kvm_iodevice_init(&dev->dev, &coalesced_mmio_ops);
 	dev->kvm = kvm;
@@ -306,17 +328,21 @@ int kvm_vm_ioctl_register_coalesced_mmio(struct kvm *kvm,
 	list_add_tail(&dev->list, &kvm->coalesced_zones);
 	mutex_unlock(&kvm->slots_lock);
 
-	return 0;
+	ret = 0;
+	goto out_free_file;
 
 out_free_dev:
 	mutex_unlock(&kvm->slots_lock);
 	kfree(dev);
+out_free_file:
+	if (use_buffer_fd)
+		fput(file);
 
 	return ret;
 }
 
 int kvm_vm_ioctl_unregister_coalesced_mmio(struct kvm *kvm,
-					   struct kvm_coalesced_mmio_zone *zone)
+					   struct kvm_coalesced_mmio_zone2 *zone)
 {
 	struct kvm_coalesced_mmio_dev *dev, *tmp;
 	int r;
virt/kvm/coalesced_mmio.h +4 -3
index d1807ce26464..32792adb7cb4 100644
@@ -19,7 +19,7 @@ struct kvm_coalesced_mmio_dev {
 	struct list_head list;
 	struct kvm_io_device dev;
 	struct kvm *kvm;
-	struct kvm_coalesced_mmio_zone zone;
+	struct kvm_coalesced_mmio_zone2 zone;
 	struct kvm_coalesced_mmio_buffer_dev *buffer_dev;
 };
 
@@ -34,9 +34,10 @@ struct kvm_coalesced_mmio_buffer_dev {
 int kvm_coalesced_mmio_init(struct kvm *kvm);
 void kvm_coalesced_mmio_free(struct kvm *kvm);
 int kvm_vm_ioctl_register_coalesced_mmio(struct kvm *kvm,
-					struct kvm_coalesced_mmio_zone *zone);
+					struct kvm_coalesced_mmio_zone2 *zone,
+					bool use_buffer_fd);
 int kvm_vm_ioctl_unregister_coalesced_mmio(struct kvm *kvm,
-					struct kvm_coalesced_mmio_zone *zone);
+					struct kvm_coalesced_mmio_zone2 *zone);
 int kvm_vm_ioctl_create_coalesced_mmio_buffer(struct kvm *kvm);
 
 #else
virt/kvm/kvm_main.c +33 -1
index 9f6ad6e03317..0850f151ef16 100644
@@ -4890,6 +4890,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 #ifdef CONFIG_KVM_MMIO
 	case KVM_CAP_COALESCED_MMIO:
 		return KVM_COALESCED_MMIO_PAGE_OFFSET;
+	case KVM_CAP_COALESCED_MMIO2:
 	case KVM_CAP_COALESCED_PIO:
 		return 1;
 #endif
@@ -5228,15 +5229,46 @@ static long kvm_vm_ioctl(struct file *filp,
 #ifdef CONFIG_KVM_MMIO
 	case KVM_REGISTER_COALESCED_MMIO: {
 		struct kvm_coalesced_mmio_zone zone;
+		struct kvm_coalesced_mmio_zone2 zone2;
 
 		r = -EFAULT;
 		if (copy_from_user(&zone, argp, sizeof(zone)))
 			goto out;
-		r = kvm_vm_ioctl_register_coalesced_mmio(kvm, &zone);
+
+		zone2.addr = zone.addr;
+		zone2.size = zone.size;
+		zone2.pio = zone.pio;
+
+		r = kvm_vm_ioctl_register_coalesced_mmio(kvm, &zone2, false);
+		break;
+	}
+	case KVM_REGISTER_COALESCED_MMIO2: {
+		struct kvm_coalesced_mmio_zone2 zone;
+
+		r = -EFAULT;
+		if (copy_from_user(&zone, argp, sizeof(zone)))
+			goto out;
+
+		r = kvm_vm_ioctl_register_coalesced_mmio(kvm, &zone, true);
 		break;
 	}
 	case KVM_UNREGISTER_COALESCED_MMIO: {
 		struct kvm_coalesced_mmio_zone zone;
+		struct kvm_coalesced_mmio_zone2 zone2;
+
+		r = -EFAULT;
+		if (copy_from_user(&zone, argp, sizeof(zone)))
+			goto out;
+
+		zone2.addr = zone.addr;
+		zone2.size = zone.size;
+		zone2.pio = zone.pio;
+
+		r = kvm_vm_ioctl_unregister_coalesced_mmio(kvm, &zone2);
+		break;
+	}
+	case KVM_UNREGISTER_COALESCED_MMIO2: {
+		struct kvm_coalesced_mmio_zone2 zone;
 
 		r = -EFAULT;
 		if (copy_from_user(&zone, argp, sizeof(zone)))
Signature · 1 line
2.34.1
#10 lore raw

PATCH v3 5/6 KVM: Documentation: Document v2 of coalesced MMIO API

In reply to Ilias Stamatis

Document the KVM_CREATE_COALESCED_MMIO_BUFFER and
KVM_REGISTER_COALESCED_MMIO2 ioctls.

---
 Documentation/virt/kvm/api.rst | 91 ++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
Documentation/virt/kvm/api.rst +91 -0
index b4d1cf2e4628..0b3ca05e380a 100644
@@ -4922,6 +4922,8 @@ For the definition of struct kvm_nested_state, see KVM_GET_NESTED_STATE.
 :Parameters: struct kvm_coalesced_mmio_zone
 :Returns: 0 on success, < 0 on error
 
+KVM_(UN)REGISTER_COALESCED_MMIO2 can be used instead if available.
+
 Coalesced I/O is a performance optimization that defers hardware
 register write emulation so that userspace exits are avoided.  It is
 typically used to reduce the overhead of emulating frequently accessed
@@ -6427,6 +6429,95 @@ the capability to be present.
 
 `flags` must currently be zero.
 
+4.144 KVM_CREATE_COALESCED_MMIO_BUFFER
+-------------------------------------
+
+:Capability: KVM_CAP_COALESCED_MMIO2
+:Architectures: all
+:Type: vm ioctl
+:Parameters: none
+:Returns: An fd on success, < 0 on error
+
+Returns an fd, but does not allocate a buffer. Also see
+KVM_REGISTER_COALESCED_MMIO2.
+
+The fd must be first passed to mmap() to allocate a page to be used as a ring
+buffer that is shared between kernel and userspace. The page must be
+interpreted as a struct kvm_coalesced_mmio_ring.
+
+::
+
+  struct kvm_coalesced_mmio_ring {
+  	__u32 first, last;
+  	struct kvm_coalesced_mmio coalesced_mmio[];
+  };
+
+The kernel will increment the last index and userspace is expected to do the
+same with the first index after consuming entries. The upper bound of the
+coalesced_mmio array is defined as KVM_COALESCED_MMIO_MAX.
+
+::
+
+  struct kvm_coalesced_mmio {
+  	__u64 phys_addr;
+  	__u32 len;
+  	union {
+  		__u32 pad;
+  		__u32 pio;
+  	};
+  	__u8  data[8];
+  };
+
+After allocating a buffer with mmap(), the fd must be passed as an argument to
+KVM_REGISTER_COALESCED_MMIO2 to associate an I/O region to which writes are
+coalesced with the ring buffer. Multiple I/O regions can be associated with the
+same ring buffer. Closing the fd after unmapping it automatically deregisters
+all I/O regions associated with it.
+
+poll() is also supported on the fd so that userspace can be notified of I/O
+writes without having to wait until the next exit to userspace.
+
+4.145 KVM_(UN)REGISTER_COALESCED_MMIO2
+-------------------------------------
+
+:Capability: KVM_CAP_COALESCED_MMIO2
+:Architectures: all
+:Type: vm ioctl
+:Parameters: struct kvm_coalesced_mmio_zone2
+:Returns: 0 on success, < 0 on error
+
+Coalesced I/O is a performance optimization that defers hardware register write
+emulation so that userspace exits are avoided. It is typically used to reduce
+the overhead of emulating frequently accessed hardware registers.
+
+When a hardware register is configured for coalesced I/O, write accesses do not
+exit to userspace and their value is recorded in a ring buffer that is shared
+between kernel and userspace.
+
+::
+
+  struct kvm_coalesced_mmio_zone2 {
+  	__u64 addr;
+  	__u32 size;
+  	union {
+  		__u32 pad;
+  		__u32 pio;
+  	};
+  	int buffer_fd;
+  };
+
+KVM_CREATE_COALESCED_MMIO_BUFFER must be used to allocate a buffer fd which
+must be first mmaped before passed to KVM_REGISTER_COALESCED_MMIO2, otherwise
+the ioctl will fail.
+
+Coalesced I/O is used if one or more write accesses to a hardware register can
+be deferred until a read or a write to another hardware register on the same
+device. This last access will cause a vmexit and userspace will process
+accesses from the ring buffer before emulating it. That will avoid exiting to
+userspace on repeated writes.
+
+Alternatively, userspace can call poll() on the buffer fd if it wishes to be
+notified of new I/O writes that way.
 
 5. The kvm_run structure
 ========================
Signature · 1 line
2.34.1
#11 lore raw

PATCH v3 6/6 KVM: selftests: Add coalesced_mmio_test

In reply to Ilias Stamatis

Test the KVM_CREATE_COALESCED_MMIO_BUFFER, KVM_REGISTER_COALESCED_MMIO2 and KVM_UNREGISTER_COALESCED_MMIO2 ioctls.

---
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/coalesced_mmio_test.c       | 313 ++++++++++++++++++
 2 files changed, 314 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/coalesced_mmio_test.c
tools/testing/selftests/kvm/Makefile +1 -0
index 48d32c5aa3eb..527297bfb9c5 100644
@@ -147,6 +147,7 @@ TEST_GEN_PROGS_x86_64 += steal_time
 TEST_GEN_PROGS_x86_64 += kvm_binary_stats_test
 TEST_GEN_PROGS_x86_64 += system_counter_offset_test
 TEST_GEN_PROGS_x86_64 += pre_fault_memory_test
+TEST_GEN_PROGS_x86_64 += coalesced_mmio_test
 
 # Compiled outputs used by test targets
 TEST_GEN_PROGS_EXTENDED_x86_64 += x86_64/nx_huge_pages_test
tools/testing/selftests/kvm/coalesced_mmio_test.c new file +313 -0

313 lines not on this page · raw message

Signature · 1 line
2.34.1
#12 lore raw

Re: [PATCH v3 6/6] KVM: selftests: Add coalesced_mmio_test

Quoted 5 lines · Ilias Stamatis

Test the KVM_CREATE_COALESCED_MMIO_BUFFER, KVM_REGISTER_COALESCED_MMIO2 and KVM_UNREGISTER_COALESCED_MMIO2 ioctls.

---

tools/testing/selftests/kvm/coalesced_mmio_test.c line 170

Quoted 9 lines
tools/testing/selftests/kvm/coalesced_mmio_test.c new file +9 -0
+	/*
+	 * Test that allocating an fd and memory mapping it works
+	 */
+	ring_fd = __vm_ioctl(vm, KVM_CREATE_COALESCED_MMIO_BUFFER, NULL);
+	TEST_ASSERT(ring_fd != -1, "Failed KVM_CREATE_COALESCED_MMIO_BUFFER");
+
+	ring = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED,
+		    ring_fd, 0);
+	TEST_ASSERT(ring != MAP_FAILED, "Failed to allocate ring buffer");

If we end up with KVM providing the buffer, there needs to be negative tests to do weird things with the mapping.

Quoted 21 lines
tools/testing/selftests/kvm/coalesced_mmio_test.c new file +21 -0
+	/*
+	 * Test that the first and last ring indices are zero
+	 */
+	TEST_ASSERT_EQ(READ_ONCE(ring->first), 0);
+	TEST_ASSERT_EQ(READ_ONCE(ring->last), 0);
+
+	/*
+	 * Run the vCPU and make sure the first MMIO write results in a
+	 * userspace exit since we have not setup MMIO coalescing yet.
+	 */
+	vcpu_run(vcpu);
+	assert_mmio_write(vcpu, MEM_REGION_GPA, MMIO_WRITE_DATA);
+
+	/*
+	 * Let's actually setup MMIO coalescing now...
+	 */
+	zone.addr = COALESCING_ZONE1_GPA;
+	zone.size = COALESCING_ZONE1_SIZE;
+	zone.buffer_fd = ring_fd;
+	r = __vm_ioctl(vm, KVM_REGISTER_COALESCED_MMIO2, &zone);
+	TEST_ASSERT(r != -1, "Failed KVM_REGISTER_COALESCED_MMIO2");

tools/testing/selftests/kvm/coalesced_mmio_test.c line 210

Quoted 10 lines
tools/testing/selftests/kvm/coalesced_mmio_test.c new file +10 -0
+
+	/*
+	 * The guest will start doing MMIO writes in the coalesced regions but
+	 * will also do a ucall when the buffer is half full. The first
+	 * userspace exit should be due to the ucall and not an MMIO exit.
+	 */
+	vcpu_run(vcpu);
+	assert_ucall_exit(vcpu, UCALL_SYNC);
+	TEST_ASSERT_EQ(READ_ONCE(ring->first), 0);
+	TEST_ASSERT_EQ(READ_ONCE(ring->last), KVM_COALESCED_MMIO_MAX / 2 + 1);

For testing the content, I generally prefer my version of the test as it has fewer magic values. To prep for this likely/potential future, I'll post a v2 that wraps the ring+mmio+pio information into a structure that can be passed around, e.g. to guest_code() and the runner+verifier. And I'll also tweak it to include the GPA/PORT in the value written, e.g. so that the test will detect if streams get crossed and a write goes to the wrong buffer.

struct kvm_coalesced_io {
	struct kvm_coalesced_mmio_ring *ring;
	uint32_t ring_size;
	uint64_t mmio_gpa;
	uint64_t *mmio;
#ifdef __x86_64__
	uint8_t pio_port;

#endif
};

That way, the basic test for multiple buffers can simply spin up two vCPUs and run them concurrently with different MMIO+PIO regions and thus different buffers.

If we want a test case that interleaves MMIO+PIO across multiple buffers on a single vCPU, it shouldn't be too hard to massage things to work with two buffers, but honestly I don't see that as being super interesting.

What would be more interesting, and probably should be added, is two vCPUs accessing the same region concurrently, e.g. to verify the locking. The test wouldn't be able to verify the order, i.e. the data can't be checked without some form of ordering in the guest code, but it'd be a good fuzzer to make sure KVM doesn't explode.