Skip to contentSkip to conversation

KVM: Bound KVM_PRE_FAULT_MEMORY by the vCPU's dirty ring

PATCH In review Fuad Tabba 3 participants 11 messages Nov 17 – Nov 19

Subsystems (7)
Stable branch Documentation x86 KVM +3 more subsystems
lore
Share & tools
lore

Fuad Tabba

author reviewer
#0 lore raw

v2 cover KVM: Bound KVM_PRE_FAULT_MEMORY by the vCPU's dirty ring

Hi folks,

Changes since v1 [1]:
- Switched the check to kvm_dirty_ring_check_request(). (Sean)
- The test checks that each page is reported exactly once, whatever the order. (Lorenzo)
- Fixed the test on 64K-page arm64 hosts by sizing its range in host pages.
- Various fixes. (Lorenzo)
- Dropped Lorenzo's Reviewed-by on patch 1, since the check changed.

KVM_PRE_FAULT_MEMORY never checks the vCPU's dirty ring, so one call over a large enough range of populated, dirty-logged pages overflows it. The first patch stops the loop at the soft limit, as KVM_RUN does. The second patch adds the case to pre_fault_memory_test.

Pre-faulting isn't the only ioctl that can push past the ring's reserve. On x86, KVM_SET_MSRS does too when it writes the wall clock or the Hyper-V hypercall page. A guest can also push past the reserve within one exit, for example through the arm64 ITS. What those paths should do when the ring fills is an open question for the dirty ring API, which I don't think belongs in this series.

I didn't include the kvm_dirty_ring_check_request() rewrite that Sean sketched on v1, since I think it can lose a request. It tests the request first and clears it afterwards. If the vCPU is scheduled out in between, loongarch sets the PV preempted flag in guest memory and x86 updates a Xen guest's runstate area, either of which pushes an entry. When that push makes the ring soft full, the request it makes is then cleared, and the vCPU runs with the ring soft full until its next push. Clearing first and re-setting, as the code does now, doesn't have that window.

Based on Linux 7.4-rc2 (7475ea8b58616; also applies to kvm-x86/next cbe54fc3462b3 with git am -3).

Cheers,
/fuad

[1] https://lore.kernel.org/all/[email protected]/

Fuad Tabba (2):
KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
KVM: selftests: Test KVM_PRE_FAULT_MEMORY against a full dirty ring

 Documentation/virt/kvm/api.rst                |   9 +-
 include/linux/kvm_dirty_ring.h                |   5 +
 .../selftests/kvm/pre_fault_memory_test.c     | 101 ++++++++++++++++++
 virt/kvm/kvm_main.c                           |   5 +
 4 files changed, 118 insertions(+), 2 deletions(-)
Signature · 1 line
2.47.3
author reviewer
#1 lore raw

PATCH v2 1/2 KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

Pre-faulting can map a dirty-logged page writable, which pushes it to the vCPU's dirty ring. KVM_RUN exits with KVM_EXIT_DIRTY_RING_FULL before the ring overflows, but the pre-fault loop never checks it, so one call can push more entries than the ring has free, which trips the WARN in kvm_dirty_ring_push() and overwrites entries userspace hasn't harvested. On x86:

  WARNING: arch/x86/kvm/../../../virt/kvm/dirty_ring.c:225 at kvm_dirty_ring_push+0x7a/0xd0
  Call Trace:
   make_spte+0x2cc/0x450
   kvm_tdp_mmu_map+0x5e1/0x7f0
   kvm_tdp_page_fault+0x12c/0x1b0
   kvm_mmu_do_page_fault+0x181/0x250
   kvm_arch_vcpu_pre_fault_memory+0x136/0x1d0
   kvm_vcpu_pre_fault_memory+0xcf/0x180
   kvm_vcpu_ioctl+0x455/0x770

Stop the loop once the ring is soft full, with the check KVM_RUN makes before entering the guest. Return the pages already processed, or -ENOSPC when there aren't any, since a retry can't succeed until userspace harvests and resets the ring. s390 has the ioctl but no ring, so give kvm_dirty_ring_check_request() a stub.

The check also sets run->exit_reason, and arm64's KVM_RUN completes a pending MMIO exit only if exit_reason is still KVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the next KVM_RUN doesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit with KVM_RUN before it pre-faults.

Fixes bc1a5cd002116 ("KVM: Add KVM_PRE_FAULT_MEMORY vcpu ioctl to pre-populate guest memory")
---
 Documentation/virt/kvm/api.rst | 9 +++++++--
 include/linux/kvm_dirty_ring.h | 5 +++++
 virt/kvm/kvm_main.c            | 5 +++++
 3 files changed, 17 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +7 -2
index e95c9cda085d9..89eb6be9bc6e3 100644
@@ -6544,6 +6544,7 @@ Errors:
   ENOENT     The specified `gpa` is outside defined memslots.
   ENOEXEC    The vCPU has not been initialised (arm64).
   EINTR      An unmasked signal is pending and no page was processed.
+  ENOSPC     The vCPU's dirty ring is full and no page was processed.
   EFAULT     The parameter address was invalid.
   EHWPOISON  A poisoned host page was encountered.
   EOPNOTSUPP Mapping memory for a GPA is unsupported by the
@@ -6588,7 +6589,9 @@ case, the ioctl can be called in parallel.
 
 When the ioctl returns, the input values are updated to point to the
 remaining range.  If `size` > 0 on return, the caller can just issue
-the ioctl again with the same `struct kvm_map_memory` argument.
+the ioctl again with the same `struct kvm_pre_fault_memory` argument.
+If that fails with ENOSPC, the vCPU's dirty ring has to be harvested
+and reset first.
 
 On x86, shadow page tables cannot support this ioctl because they
 are indexed by virtual address or nested guest physical address.
@@ -8957,7 +8960,9 @@ Therefore, the ioctl must be called *before* reading the content of
 the dirty pages.
 
 The dirty ring can get full.  When it happens, the KVM_RUN of the
-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL.
+vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and
+KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the
+ring has been harvested and reset.
 
 The dirty ring interface has a major difference comparing to the
 KVM_GET_DIRTY_LOG interface in that, when reading the dirty ring from
include/linux/kvm_dirty_ring.h +5 -0
index eb10d87adf7d5..729615ae0cf4d 100644
@@ -60,6 +60,11 @@ static inline void kvm_dirty_ring_push(struct kvm_vcpu *vcpu,
 {
 }
 
+static inline bool kvm_dirty_ring_check_request(struct kvm_vcpu *vcpu)
+{
+	return false;
+}
+
 static inline struct page *kvm_dirty_ring_get_page(struct kvm_dirty_ring *ring,
 						   u32 offset)
 {
virt/kvm/kvm_main.c +5 -0
index cf4cb586f1407..ac8b94a386e4a 100644
@@ -4461,6 +4461,11 @@ static int kvm_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
 			break;
 		}
 
+		if (kvm_dirty_ring_check_request(vcpu)) {
+			r = -ENOSPC;
+			break;
+		}
+
 		r = kvm_arch_vcpu_pre_fault_memory(vcpu, range);
 		if (WARN_ON_ONCE(r == 0 || r == -EIO))
 			break;
Signature · 1 line
2.47.3
author reviewer
#9 lore raw

PATCH v2 2/2 KVM: selftests: Test KVM_PRE_FAULT_MEMORY against a full dirty ring

In reply to Fuad Tabba

Pre-fault more populated, dirty-logged pages than the vCPU's dirty ring holds, and check that each call stops before the ring overflows, that a retry returns ENOSPC without progress, and that each page is reported exactly once. A read pre-fault can map a page read-only, or access-tracked on x86 without EPT A/D bits, so skip the case if pre-faulting doesn't dirty any page.

---
 .../selftests/kvm/pre_fault_memory_test.c     | 101 ++++++++++++++++++
 1 file changed, 101 insertions(+)
tools/testing/selftests/kvm/pre_fault_memory_test.c +101 -0
index 3082fe09b95fa..72c501afc873b 100644
@@ -5,6 +5,9 @@
  * Author:
  * Isaku Yamahata <isaku.yamahata at gmail.com>
  */
+#include <asm/barrier.h>
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
 #include <linux/sizes.h>
 
 #include <test_util.h>
@@ -17,6 +20,8 @@
 /* Arbitrarily chosen values */
 #define TEST_BASE_SIZE		SZ_2M
 #define TEST_SLOT		10
+/* Large enough for the entries every architecture reserves. */
+#define TEST_RING_SIZE		SZ_64K
 
 /* Storage of test info to share with guest code */
 struct test_config {
@@ -281,6 +286,101 @@ static void test_pre_fault_memory(unsigned long vm_type, enum vm_mem_backing_src
 	for_each_guest_mode(__test_pre_fault_memory, &p);
 }
 
+/*
+ * Pre-fault more dirty-logged pages than the ring holds: each call has to stop
+ * before the ring overflows, and no entry may be lost.
+ */
+static void test_pre_fault_dirty_ring(void)
+{
+	const u64 entries = TEST_RING_SIZE / sizeof(struct kvm_dirty_gfn);
+	const u64 npages = entries + entries / 4;
+	struct kvm_pre_fault_memory range, left;
+	struct kvm_dirty_gfn *gfns, *cur;
+	u64 harvested = 0, size;
+	struct kvm_vcpu *vcpu;
+	unsigned long *seen;
+	int ret, stops = 0;
+	struct kvm_vm *vm;
+	u32 fetch = 0;
+	gpa_t gpa;
+
+	if (!kvm_has_cap(KVM_CAP_DIRTY_LOG_RING) && !kvm_has_cap(KVM_CAP_DIRTY_LOG_RING_ACQ_REL)) {
+		pr_info("Skipping the dirty ring test: no dirty ring\n");
+		return;
+	}
+
+	vm = vm_create(1);
+	vm_enable_dirty_ring(vm, TEST_RING_SIZE);
+	vcpu = vm_vcpu_add(vm, 0, NULL);
+	gfns = vcpu_map_dirty_ring(vcpu);
+
+	/* A ring entry is a host page. */
+	size = npages * getpagesize();
+	gpa = align_down(vm->max_gfn * vm->page_size - size, SZ_2M);
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, gpa, TEST_SLOT,
+				    vm_num_guest_pages(vm->mode, npages), KVM_MEM_LOG_DIRTY_PAGES);
+
+	/* Populate the pages, so that pre-faulting can map them writable. */
+	memset(addr_gpa2hva(vm, gpa), 1, size);
+
+	seen = bitmap_zalloc(npages);
+	TEST_ASSERT(seen, "Failed to allocate the bitmap of reported pages");
+
+	range = (struct kvm_pre_fault_memory) { .gpa = gpa, .size = size };
+	while (range.size) {
+		u64 n;
+
+		vcpu_ioctl(vcpu, KVM_PRE_FAULT_MEMORY, &range);
+
+		if (range.size) {
+			left = range;
+			ret = __vcpu_ioctl(vcpu, KVM_PRE_FAULT_MEMORY, &range);
+			TEST_ASSERT_VM_VCPU_IOCTL(ret == -1 && errno == ENOSPC,
+						  KVM_PRE_FAULT_MEMORY, ret, vm);
+			TEST_ASSERT(range.gpa == left.gpa && range.size == left.size,
+				    "KVM_PRE_FAULT_MEMORY made progress on a full ring");
+			stops++;
+		}
+
+		for (n = 0; ; n++) {
+			cur = &gfns[fetch % entries];
+			/* Pairs with the smp_wmb() in kvm_dirty_ring_push(). */
+			if (smp_load_acquire(&cur->flags) != KVM_DIRTY_GFN_F_DIRTY)
+				break;
+
+			TEST_ASSERT(cur->slot == TEST_SLOT && cur->offset < npages,
+				    "Ring entry %u: slot %u offset 0x%llx, expected slot %u offset < 0x%lx",
+				    fetch, cur->slot, cur->offset, TEST_SLOT, npages);
+			TEST_ASSERT(!__test_and_set_bit(cur->offset, seen),
+				    "Ring entry %u: offset 0x%llx reported twice",
+				    fetch, cur->offset);
+			/* Pairs with kvm_dirty_gfn_harvested()'s smp_load_acquire(). */
+			smp_store_release(&cur->flags, KVM_DIRTY_GFN_F_RESET);
+			fetch++;
+		}
+
+		/* A read pre-fault can map a page read-only, or access-tracked on x86. */
+		if (!n && !harvested && !range.size) {
+			pr_info("Skipping the dirty ring test: pre-faulting dirtied no pages\n");
+			bitmap_free(seen);
+			kvm_vm_free(vm);
+			return;
+		}
+		TEST_ASSERT(n, "KVM_PRE_FAULT_MEMORY pushed nothing to the ring");
+		TEST_ASSERT(kvm_vm_reset_dirty_ring(vm) == n,
+			    "KVM_RESET_DIRTY_RINGS didn't reset %lu entries", n);
+		harvested += n;
+	}
+
+	TEST_ASSERT(harvested == npages, "%lu ring entries for %lu pages", harvested, npages);
+	TEST_ASSERT(stops, "KVM_PRE_FAULT_MEMORY never stopped for a %lu-entry ring", entries);
+	pr_info("Pre-faulted %lu pages through a %lu-entry ring, stopping %d times\n",
+		npages, entries, stops);
+
+	bitmap_free(seen);
+	kvm_vm_free(vm);
+}
+
 static void help(char *name)
 {
 	puts("");
@@ -320,5 +420,6 @@ int main(int argc, char *argv[])
 	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, false);
 	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, true);
 #endif
+	test_pre_fault_dirty_ring();
 	return 0;
 }
Signature · 1 line
2.47.3
#2 lore raw

Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

In reply to Fuad Tabba
Quoted 23 lines · Fuad Tabba

Pre-faulting can map a dirty-logged page writable, which pushes it to the vCPU's dirty ring. KVM_RUN exits with KVM_EXIT_DIRTY_RING_FULL before the ring overflows, but the pre-fault loop never checks it, so one call can push more entries than the ring has free, which trips the WARN in kvm_dirty_ring_push() and overwrites entries userspace hasn't harvested. On x86:

  WARNING: arch/x86/kvm/../../../virt/kvm/dirty_ring.c:225 at kvm_dirty_ring_push+0x7a/0xd0
  Call Trace:
   make_spte+0x2cc/0x450
   kvm_tdp_mmu_map+0x5e1/0x7f0
   kvm_tdp_page_fault+0x12c/0x1b0
   kvm_mmu_do_page_fault+0x181/0x250
   kvm_arch_vcpu_pre_fault_memory+0x136/0x1d0
   kvm_vcpu_pre_fault_memory+0xcf/0x180
   kvm_vcpu_ioctl+0x455/0x770

Stop the loop once the ring is soft full, with the check KVM_RUN makes before entering the guest. Return the pages already processed, or -ENOSPC when there aren't any, since a retry can't succeed until userspace harvests and resets the ring. s390 has the ioctl but no ring, so give kvm_dirty_ring_check_request() a stub.

Quoted 6 lines

The check also sets run->exit_reason, and arm64's KVM_RUN completes a pending MMIO exit only if exit_reason is still KVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the next KVM_RUN doesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit with KVM_RUN before it pre-faults.

That's also true for riscv:

	switch (run->exit_reason) {
	case KVM_EXIT_MMIO:
		/* Process MMIO value returned from user-space */
		ret = kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run);
		break;
	case KVM_EXIT_RISCV_SBI:
		/* Process SBI value returned from user-space */
		ret = kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run);
		break;
	case KVM_EXIT_RISCV_CSR:
		/* Process CSR value returned from user-space */
		ret = kvm_riscv_vcpu_csr_return(vcpu, vcpu->run);
		break;
	}

(note that it skips CSR, SBI also with a changed exit_reason).

But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?

Quoted 34 lines
Fixes bc1a5cd002116 ("KVM: Add KVM_PRE_FAULT_MEMORY vcpu ioctl to pre-populate guest memory")
---
 Documentation/virt/kvm/api.rst | 9 +++++++--
 include/linux/kvm_dirty_ring.h | 5 +++++
 virt/kvm/kvm_main.c            | 5 +++++
 3 files changed, 17 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +4 -1
index e95c9cda085d9..89eb6be9bc6e3 100644
@@ -6544,6 +6544,7 @@ Errors:
   ENOENT     The specified `gpa` is outside defined memslots.
   ENOEXEC    The vCPU has not been initialised (arm64).
   EINTR      An unmasked signal is pending and no page was processed.
+  ENOSPC     The vCPU's dirty ring is full and no page was processed.
   EFAULT     The parameter address was invalid.
   EHWPOISON  A poisoned host page was encountered.
   EOPNOTSUPP Mapping memory for a GPA is unsupported by the
@@ -6588,7 +6589,9 @@ case, the ioctl can be called in parallel.
 
 When the ioctl returns, the input values are updated to point to the
 remaining range.  If `size` > 0 on return, the caller can just issue
-the ioctl again with the same `struct kvm_map_memory` argument.
+the ioctl again with the same `struct kvm_pre_fault_memory` argument.
+If that fails with ENOSPC, the vCPU's dirty ring has to be harvested
+and reset first.
 
 On x86, shadow page tables cannot support this ioctl because they
 are indexed by virtual address or nested guest physical address.

Documentation/virt/kvm/api.rst Therefore, the ioctl must be called *before* reading the content of · line 8965

Quoted 7 lines
Documentation/virt/kvm/api.rst +3 -1
@@ -8957,7 +8960,9 @@ Therefore, the ioctl must be called *before* reading the content of
 the dirty pages.
 
 The dirty ring can get full.  When it happens, the KVM_RUN of the
-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL.
+vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and
+KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the
+ring has been harvested and reset.

Maybe worth mentioning the exit_reason thing if we keep this arch-generic?

Quoted 36 lines
Documentation/virt/kvm/api.rst +0 -0
 The dirty ring interface has a major difference comparing to the
 KVM_GET_DIRTY_LOG interface in that, when reading the dirty ring from
include/linux/kvm_dirty_ring.h +5 -0
index eb10d87adf7d5..729615ae0cf4d 100644
@@ -60,6 +60,11 @@ static inline void kvm_dirty_ring_push(struct kvm_vcpu *vcpu,
 {
 }
 
+static inline bool kvm_dirty_ring_check_request(struct kvm_vcpu *vcpu)
+{
+	return false;
+}
+
 static inline struct page *kvm_dirty_ring_get_page(struct kvm_dirty_ring *ring,
 						   u32 offset)
 {
virt/kvm/kvm_main.c +5 -1
index cf4cb586f1407..ac8b94a386e4a 100644
@@ -4461,6 +4461,11 @@ static int kvm_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
 			break;
 		}
 
+		if (kvm_dirty_ring_check_request(vcpu)) {
+			r = -ENOSPC;
+			break;
+		}
+
 		r = kvm_arch_vcpu_pre_fault_memory(vcpu, range);
 		if (WARN_ON_ONCE(r == 0 || r == -EIO))
 			break;
--
Quoted 2 lines

2.47.3

I wonder if this shouldn't just be an x86-only fix, since no other arch is impacted AFAICT.

arm64 and riscv never map a dirty-logged memslot writable, so they're never pushed to the rin, and s390 doesn't have a dirty ring at all:

arm64:

static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd,
const struct kvm_s2_fault_vma_info *s2vi, enum kvm_pgtable_prot *prot)

{
	...
	if (s2vi->map_writable && (s2vi->device ||
				   !memslot_is_logging(s2fd->memslot) ||
				   kvm_s2_fault_is_write(s2fd)))
		*prot |= KVM_PGTABLE_PROT_W;
	...
}

riscv:

int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
gpa_t gpa, unsigned long hva, bool is_write, struct kvm_gstage_mapping *out_map)

{
	...
	bool logging = kvm_slot_dirty_track_enabled(memslot) &&
		       !(memslot->flags & KVM_MEM_READONLY);
	...
	/*
	 * If logging is active then we allow writable pages only
	 * for write faults.
	 */
	if (logging && !is_write)
		writable = false;
	...
}

An x86-only version would be pretty simple, something like the below.

This also covers off the retry (RET_PF_RETRY) loop which a generic version doesn't.

arch/x86/kvm/mmu/mmu.c +3 -0
index f89e9f8a6cc2..343eb5ad0baa 100644
@@ -5129,6 +5129,14 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa,
 		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
 			return -EIO;
 
+		if (kvm_dirty_ring_check_request(vcpu))
+			return -ENOSPC;
+
 		r = kvm_mmu_reload(vcpu);
 		if (r)
 			return r;

--
Cheers, Lorenzo

Acked-by on v2 2/2
#10 lore raw

Re: [PATCH v2 2/2] KVM: selftests: Test KVM_PRE_FAULT_MEMORY against a full dirty ring

In reply to Fuad Tabba
Quoted 8 lines · Fuad Tabba

Pre-fault more populated, dirty-logged pages than the vCPU's dirty ring holds, and check that each call stops before the ring overflows, that a retry returns ENOSPC without progress, and that each page is reported exactly once. A read pre-fault can map a page read-only, or access-tracked on x86 without EPT A/D bits, so skip the case if pre-faulting doesn't dirty any page.

LGTM!

Quoted 137 lines
---
 .../selftests/kvm/pre_fault_memory_test.c     | 101 ++++++++++++++++++
 1 file changed, 101 insertions(+)
tools/testing/selftests/kvm/pre_fault_memory_test.c +101 -1
index 3082fe09b95fa..72c501afc873b 100644
@@ -5,6 +5,9 @@
  * Author:
  * Isaku Yamahata <isaku.yamahata at gmail.com>
  */
+#include <asm/barrier.h>
+#include <linux/bitmap.h>
+#include <linux/bitops.h>
 #include <linux/sizes.h>
 
 #include <test_util.h>
@@ -17,6 +20,8 @@
 /* Arbitrarily chosen values */
 #define TEST_BASE_SIZE		SZ_2M
 #define TEST_SLOT		10
+/* Large enough for the entries every architecture reserves. */
+#define TEST_RING_SIZE		SZ_64K
 
 /* Storage of test info to share with guest code */
 struct test_config {
@@ -281,6 +286,101 @@ static void test_pre_fault_memory(unsigned long vm_type, enum vm_mem_backing_src
 	for_each_guest_mode(__test_pre_fault_memory, &p);
 }
 
+/*
+ * Pre-fault more dirty-logged pages than the ring holds: each call has to stop
+ * before the ring overflows, and no entry may be lost.
+ */
+static void test_pre_fault_dirty_ring(void)
+{
+	const u64 entries = TEST_RING_SIZE / sizeof(struct kvm_dirty_gfn);
+	const u64 npages = entries + entries / 4;
+	struct kvm_pre_fault_memory range, left;
+	struct kvm_dirty_gfn *gfns, *cur;
+	u64 harvested = 0, size;
+	struct kvm_vcpu *vcpu;
+	unsigned long *seen;
+	int ret, stops = 0;
+	struct kvm_vm *vm;
+	u32 fetch = 0;
+	gpa_t gpa;
+
+	if (!kvm_has_cap(KVM_CAP_DIRTY_LOG_RING) && !kvm_has_cap(KVM_CAP_DIRTY_LOG_RING_ACQ_REL)) {
+		pr_info("Skipping the dirty ring test: no dirty ring\n");
+		return;
+	}
+
+	vm = vm_create(1);
+	vm_enable_dirty_ring(vm, TEST_RING_SIZE);
+	vcpu = vm_vcpu_add(vm, 0, NULL);
+	gfns = vcpu_map_dirty_ring(vcpu);
+
+	/* A ring entry is a host page. */
+	size = npages * getpagesize();
+	gpa = align_down(vm->max_gfn * vm->page_size - size, SZ_2M);
+	vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, gpa, TEST_SLOT,
+				    vm_num_guest_pages(vm->mode, npages), KVM_MEM_LOG_DIRTY_PAGES);
+
+	/* Populate the pages, so that pre-faulting can map them writable. */
+	memset(addr_gpa2hva(vm, gpa), 1, size);
+
+	seen = bitmap_zalloc(npages);
+	TEST_ASSERT(seen, "Failed to allocate the bitmap of reported pages");
+
+	range = (struct kvm_pre_fault_memory) { .gpa = gpa, .size = size };
+	while (range.size) {
+		u64 n;
+
+		vcpu_ioctl(vcpu, KVM_PRE_FAULT_MEMORY, &range);
+
+		if (range.size) {
+			left = range;
+			ret = __vcpu_ioctl(vcpu, KVM_PRE_FAULT_MEMORY, &range);
+			TEST_ASSERT_VM_VCPU_IOCTL(ret == -1 && errno == ENOSPC,
+						  KVM_PRE_FAULT_MEMORY, ret, vm);
+			TEST_ASSERT(range.gpa == left.gpa && range.size == left.size,
+				    "KVM_PRE_FAULT_MEMORY made progress on a full ring");
+			stops++;
+		}
+
+		for (n = 0; ; n++) {
+			cur = &gfns[fetch % entries];
+			/* Pairs with the smp_wmb() in kvm_dirty_ring_push(). */
+			if (smp_load_acquire(&cur->flags) != KVM_DIRTY_GFN_F_DIRTY)
+				break;
+
+			TEST_ASSERT(cur->slot == TEST_SLOT && cur->offset < npages,
+				    "Ring entry %u: slot %u offset 0x%llx, expected slot %u offset < 0x%lx",
+				    fetch, cur->slot, cur->offset, TEST_SLOT, npages);
+			TEST_ASSERT(!__test_and_set_bit(cur->offset, seen),
+				    "Ring entry %u: offset 0x%llx reported twice",
+				    fetch, cur->offset);
+			/* Pairs with kvm_dirty_gfn_harvested()'s smp_load_acquire(). */
+			smp_store_release(&cur->flags, KVM_DIRTY_GFN_F_RESET);
+			fetch++;
+		}
+
+		/* A read pre-fault can map a page read-only, or access-tracked on x86. */
+		if (!n && !harvested && !range.size) {
+			pr_info("Skipping the dirty ring test: pre-faulting dirtied no pages\n");
+			bitmap_free(seen);
+			kvm_vm_free(vm);
+			return;
+		}
+		TEST_ASSERT(n, "KVM_PRE_FAULT_MEMORY pushed nothing to the ring");
+		TEST_ASSERT(kvm_vm_reset_dirty_ring(vm) == n,
+			    "KVM_RESET_DIRTY_RINGS didn't reset %lu entries", n);
+		harvested += n;
+	}
+
+	TEST_ASSERT(harvested == npages, "%lu ring entries for %lu pages", harvested, npages);
+	TEST_ASSERT(stops, "KVM_PRE_FAULT_MEMORY never stopped for a %lu-entry ring", entries);
+	pr_info("Pre-faulted %lu pages through a %lu-entry ring, stopping %d times\n",
+		npages, entries, stops);
+
+	bitmap_free(seen);
+	kvm_vm_free(vm);
+}
+
 static void help(char *name)
 {
 	puts("");
@@ -320,5 +420,6 @@ int main(int argc, char *argv[])
 	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, false);
 	test_pre_fault_memory(KVM_X86_SW_PROTECTED_VM, backing, true);
 #endif
+	test_pre_fault_dirty_ring();
 	return 0;
 }
--
Quoted 2 lines

2.47.3

--
Cheers, Lorenzo

author reviewer
#3 lore raw

Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

In reply to Lorenzo Stoakes (ARM)

Hi Lorenzo,

On Tue, 17 Nov 2026 14:23:09 +0000, "Lorenzo Stoakes (ARM)" <[email protected]> wrote:
[...]

Quoted 24 lines
Quoted 6 lines

The check also sets run->exit_reason, and arm64's KVM_RUN completes a pending MMIO exit only if exit_reason is still KVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the next KVM_RUN doesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit with KVM_RUN before it pre-faults.

That's also true for riscv:

        switch (run->exit_reason) {
        case KVM_EXIT_MMIO:
                /* Process MMIO value returned from user-space */
                ret = kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run);
                break;
        case KVM_EXIT_RISCV_SBI:
                /* Process SBI value returned from user-space */
                ret = kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run);
                break;
        case KVM_EXIT_RISCV_CSR:
                /* Process CSR value returned from user-space */
                ret = kvm_riscv_vcpu_csr_return(vcpu, vcpu->run);
                break;
        }
Quoted 1 line

(note that it skips CSR, SBI also with a changed exit_reason).

Ack.

Quoted 2 lines

But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?

The ring overflows on arm64 too (below). As for exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between a KVM_EXIT_MMIO return and the next KVM_RUN "gets to deal with the fallout" [1]. I think the same goes for riscv's exits.

[...]

Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0 -0

[...]

Quoted 7 lines

Documentation/virt/kvm/api.rst

Quoted 5 lines

The dirty ring can get full. When it happens, the KVM_RUN of the

Documentation/virt/kvm/api.rst +3 -1
-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL.
+vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and
+KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the
+ring has been harvested and reset.

Maybe worth mentioning the exit_reason thing if we keep this arch-generic?

Sure, I'll document that userspace should complete a pending exit with KVM_RUN before it pre-faults on that vCPU.

[...]

Quoted 11 lines

I wonder if this shouldn't just be an x86-only fix, since no other arch is impacted AFAICT.

arm64 and riscv never map a dirty-logged memslot writable, so they're never pushed to the rin, and s390 doesn't have a dirty ring at all:

arm64:

static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd,
const struct kvm_s2_fault_vma_info *s2vi, enum kvm_pgtable_prot *prot)

Quoted 8 lines
{
        ...
        if (s2vi->map_writable && (s2vi->device ||
                                   !memslot_is_logging(s2fd->memslot) ||
                                   kvm_s2_fault_is_write(s2fd)))
                *prot |= KVM_PGTABLE_PROT_W;
        ...
}

arm64 does map a logged slot as writable today. memslot_is_logging() tests memslot->dirty_bitmap, which isn't allocated when only the dirty ring is enabled, and patch 2's selftest triggers the WARN in kvm_dirty_ring_push() on arm64 without patch 1. Wei-Lin's fix to memslot_is_logging() [2] is in kvmarm/fixes, not in mainline yet.

With it, the s2vi->device case above still maps a device page writable on a read fault in a logged slot. kvm_arch_prepare_memory_region() rejects logging on a VM_PFNMAP VMA, so that requires userspace to map the device under the slot after creating it (I haven't tested it).

So I think the check should be in the generic loop.

[...]

Quoted 4 lines

An x86-only version would be pretty simple, something like the below.

This also covers off the retry (RET_PF_RETRY) loop which a generic version doesn't.

True. To overflow there, the retries for one page would have to push more entries than the ring has in reserve (at least 64), and I don't see a way to get that, so I left it alone.

Cheers,
/fuad

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

#4 lore raw

Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

Quoted 20 lines · Fuad Tabba
Quoted 2 lines

But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?

The ring overflows on arm64 too (below). As for exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between a KVM_EXIT_MMIO return and the next KVM_RUN "gets to deal with the fallout" [1]. I think the same goes for riscv's exits.

[...]

Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0 -0

[...]

Quoted 7 lines

Documentation/virt/kvm/api.rst

Quoted 5 lines

The dirty ring can get full. When it happens, the KVM_RUN of the

Documentation/virt/kvm/api.rst +3 -1
-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL.
+vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and
+KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the
+ring has been harvested and reset.

Maybe worth mentioning the exit_reason thing if we keep this arch-generic?

Quoted 2 lines

Sure, I'll document that userspace should complete a pending exit with KVM_RUN before it pre-faults on that vCPU.

If we're going to do a documentation update, I vote to keep it vague and scary, otherwise readers might think the exception proves the rule. I.e. if we say that KVM_PRE_FAULT_MEMORY isn't supported, then userspace might think all other ioctls are safe, which is most definitely not the case. E.g. something like

Documentation/virt/kvm/api.rst +7 -4
index e95c9cda085d..45dc1e3c0b03 100644
@@ -6971,10 +6971,13 @@ to the byte array.
 
       The pending state of the operation is not preserved in state which is
       visible to userspace, thus userspace should ensure that the operation is
-      completed before performing a live migration.  Userspace can re-enter the
-      guest with an unmasked signal pending or with the immediate_exit field set
-      to complete pending operations without allowing any further instructions
-      to be executed.
+      completed before performing a live migration.  In general, userspace should
+      exercise extreme caution when invoking ioctls other than KVM_RUN prior to
+      completing the original operation.
+
+      Userspace can re-enter the guest with an unmasked signal pending or with
+      the immediate_exit field set to complete pending operations without
+      allowing any further instructions to be executed.
 
 ::
author reviewer
#5 lore raw

Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

Hi Sean,

On Thu, 19 Nov 2026 at 00:25, Sean Christopherson <[email protected]> wrote:
[...]

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Maybe worth mentioning the exit_reason thing if we keep this arch-generic?

Sure, I'll document that userspace should complete a pending exit with KVM_RUN before it pre-faults on that vCPU.

If we're going to do a documentation update, I vote to keep it vague and scary, otherwise readers might think the exception proves the rule. I.e. if we say that KVM_PRE_FAULT_MEMORY isn't supported, then userspace might think all other ioctls are safe, which is most definitely not the case. E.g. something like

I'll go with that in v3.

Thanks,
/fuad

#8 lore raw

Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

In reply to Fuad Tabba
Quoted 29 lines · Fuad Tabba

Hi Lorenzo,

On Tue, 17 Nov 2026 14:23:09 +0000, "Lorenzo Stoakes (ARM)" <[email protected]> wrote:
[...]

Quoted 24 lines
Quoted 6 lines

The check also sets run->exit_reason, and arm64's KVM_RUN completes a pending MMIO exit only if exit_reason is still KVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the next KVM_RUN doesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit with KVM_RUN before it pre-faults.

That's also true for riscv:

        switch (run->exit_reason) {
        case KVM_EXIT_MMIO:
                /* Process MMIO value returned from user-space */
                ret = kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run);
                break;
        case KVM_EXIT_RISCV_SBI:
                /* Process SBI value returned from user-space */
                ret = kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run);
                break;
        case KVM_EXIT_RISCV_CSR:
                /* Process CSR value returned from user-space */
                ret = kvm_riscv_vcpu_csr_return(vcpu, vcpu->run);
                break;
        }
Quoted 3 lines
Quoted 1 line

(note that it skips CSR, SBI also with a changed exit_reason).

Ack.

Probably worth referencing in commit msg I guess?

Quoted 9 lines
Quoted 3 lines

But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?

The ring overflows on arm64 too (below). As for exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between a KVM_EXIT_MMIO return and the next KVM_RUN "gets to deal with the fallout" [1]. I think the same goes for riscv's exits.

I mean generally yeah, users doing silly things can expect silly results I guess! :)

Quoted 12 lines

[...]

Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0 -0

[...]

Quoted 7 lines

Documentation/virt/kvm/api.rst

Quoted 5 lines

The dirty ring can get full. When it happens, the KVM_RUN of the

Documentation/virt/kvm/api.rst +3 -1
-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL.
+vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and
+KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the
+ring has been harvested and reset.

Maybe worth mentioning the exit_reason thing if we keep this arch-generic?

Quoted 2 lines

Sure, I'll document that userspace should complete a pending exit with KVM_RUN before it pre-faults on that vCPU.

Thanks!

Quoted 22 lines

[...]

Quoted 11 lines

I wonder if this shouldn't just be an x86-only fix, since no other arch is impacted AFAICT.

arm64 and riscv never map a dirty-logged memslot writable, so they're never pushed to the rin, and s390 doesn't have a dirty ring at all:

arm64:

static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd,
const struct kvm_s2_fault_vma_info *s2vi, enum kvm_pgtable_prot *prot)

Quoted 8 lines
{
        ...
        if (s2vi->map_writable && (s2vi->device ||
                                   !memslot_is_logging(s2fd->memslot) ||
                                   kvm_s2_fault_is_write(s2fd)))
                *prot |= KVM_PGTABLE_PROT_W;
        ...
}
Quoted 5 lines

arm64 does map a logged slot as writable today. memslot_is_logging() tests memslot->dirty_bitmap, which isn't allocated when only the dirty ring is enabled, and patch 2's selftest triggers the WARN in kvm_dirty_ring_push() on arm64 without patch 1. Wei-Lin's fix to memslot_is_logging() [2] is in kvmarm/fixes, not in mainline yet.

But Wei-Lin's fix was taken? So I don't think predicating things on the basis of it not being there really makes any sense.

However...

Quoted 5 lines

With it, the s2vi->device case above still maps a device page writable on a read fault in a logged slot. kvm_arch_prepare_memory_region() rejects logging on a VM_PFNMAP VMA, so that requires userspace to map the device under the slot after creating it (I haven't tested it).

...Given that this possibility exists and device pages are unconditionally marked writable (even if it's mildly contrived to get there), it's fair enough to put it in the generic code.

Quoted 4 lines

So I think the check should be in the generic loop.

[...]

Quoted 8 lines
Quoted 4 lines

An x86-only version would be pretty simple, something like the below.

This also covers off the retry (RET_PF_RETRY) loop which a generic version doesn't.

True. To overflow there, the retries for one page would have to push more entries than the ring has in reserve (at least 64), and I don't see a way to get that, so I left it alone.

Yep that's fair enough!

Quoted 6 lines

Cheers,
/fuad

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

--
Cheers, Lorenzo

#6 lore raw

Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

In reply to Sean Christopherson
Quoted 21 lines · Sean Christopherson
Quoted 20 lines · Fuad Tabba
Quoted 2 lines

But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?

The ring overflows on arm64 too (below). As for exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between a KVM_EXIT_MMIO return and the next KVM_RUN "gets to deal with the fallout" [1]. I think the same goes for riscv's exits.

[...]

Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0 -0

[...]

Quoted 7 lines

Documentation/virt/kvm/api.rst

Quoted 5 lines

The dirty ring can get full. When it happens, the KVM_RUN of the

Documentation/virt/kvm/api.rst +3 -1
-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL.
+vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and
+KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the
+ring has been harvested and reset.

Maybe worth mentioning the exit_reason thing if we keep this arch-generic?

Quoted 7 lines
Quoted 2 lines

Sure, I'll document that userspace should complete a pending exit with KVM_RUN before it pre-faults on that vCPU.

If we're going to do a documentation update, I vote to keep it vague and scary, otherwise readers might think the exception proves the rule. I.e. if we say that KVM_PRE_FAULT_MEMORY isn't supported, then userspace might think all other ioctls are safe, which is most definitely not the case. E.g. something like

Yes definitely, like drivers, users can get up to all sorts of mischief if not properly forewarned :>)

Quoted 12 lines
Documentation/virt/kvm/api.rst +0 -3
index e95c9cda085d..45dc1e3c0b03 100644
@@ -6971,10 +6971,13 @@ to the byte array.
 
       The pending state of the operation is not preserved in state which is
       visible to userspace, thus userspace should ensure that the operation is
-      completed before performing a live migration.  Userspace can re-enter the
-      guest with an unmasked signal pending or with the immediate_exit field set
-      to complete pending operations without allowing any further instructions

Documentation/virt/kvm/api.rst to the byte array. · line 6982

Quoted 10 lines
Documentation/virt/kvm/api.rst +7 -1
-      to be executed.
+      completed before performing a live migration.  In general, userspace should
+      exercise extreme caution when invoking ioctls other than KVM_RUN prior to
+      completing the original operation.
+
+      Userspace can re-enter the guest with an unmasked signal pending or with
+      the immediate_exit field set to complete pending operations without
+      allowing any further instructions to be executed.
 
 ::

--
Cheers, Lorenzo

author reviewer
#7 lore raw

Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full

Hi Lorenzo,

On Thu, 19 Nov 2026 11:18:48 +0000, "Lorenzo Stoakes (ARM)" <[email protected]> wrote:
[...]

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

(note that it skips CSR, SBI also with a changed exit_reason).

Ack.

Probably worth referencing in commit msg I guess?

Will add one in v3.

Cheers,
/fuad