Skip to contentSkip to conversation

KVM: riscv: Add KVM_PRE_FAULT_MEMORY support

PATCH Superseded Jinyu Tang 2 participants 8 messages Aug 15

Subsystems (4)
lore
Share & tools
lore

Jinyu Tang

author
#0 lore raw

v4 cover KVM: riscv: Add KVM_PRE_FAULT_MEMORY support

Add RISC-V support for the generic KVM_PRE_FAULT_MEMORY ioctl.

Patch 1 and patch 2 fix existing G-stage huge-page corner cases. They are included first because the prefault hook relies on the map path reporting the mapping that actually covers the requested GPA.

Patch 3 adds the RISC-V arch hook for KVM_PRE_FAULT_MEMORY. Patch 4 adds Sv57 indexing support to the RISC-V KVM selftest helpers, and patch 5 enables the generic pre_fault_memory_test for RISC-V.

Patch 1 folds the previously posted standalone G-stage table overwrite fix into this series:

https://lore.kernel.org/kvm-riscv/[email protected]/T/#u

Changes in v4:
- Fold the standalone G-stage non-leaf overwrite fix into this series.
- Avoid THP adjustment when the original faulting GPA already has an existing 4K G-stage leaf.
- Return an error for HWPOISON instead of reporting success without a visible G-stage mapping. (Sashiko)

Changes in v3:
- Retry internally when the map path returns success without a visible G-stage mapping, instead of exposing -EAGAIN. (Sashiko)

Changes in v2:
- Drop the per-test guest_modes_append_default() call from pre_fault_memory_test.c. (Sean)

Jinyu Tang (5):
KVM: riscv: Avoid overwriting G-stage tables with huge leaves
KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves
KVM: riscv: Implement KVM_PRE_FAULT_MEMORY
KVM: selftests: Add RISC-V Sv57 page table indexing
KVM: selftests: Enable pre_fault_memory_test for RISC-V

 arch/riscv/kvm/Kconfig                        |  1 +
 arch/riscv/kvm/gstage.c                       |  9 ++
 arch/riscv/kvm/mmu.c                          | 83 ++++++++++++++++++-
 arch/riscv/kvm/vm.c                           |  1 +
 tools/testing/selftests/kvm/Makefile.kvm      |  1 +
 .../selftests/kvm/include/riscv/processor.h   |  6 ++
 .../selftests/kvm/lib/riscv/processor.c       |  2 +
 7 files changed, 100 insertions(+), 3 deletions(-)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v4 1/5 KVM: riscv: Avoid overwriting G-stage tables with huge leaves

RISC-V KVM can overwrite an existing G-stage table entry when installing a huge leaf mapping. If the target huge range already has a lower-level page table, kvm_riscv_gstage_set_pte() can replace the non-leaf entry with a leaf PTE and disconnect the lower-level page table.

Reject replacing a valid table entry with a leaf PTE. If huge-page installation hits such a conflict, fall back to a 4K mapping for the original faulting GPA in the MMU fault path, where the original GPA and HFN are still available.

Fixes 9d05c1fee837 ("RISC-V: KVM: Implement stage2 page table programming")
---
 arch/riscv/kvm/gstage.c |  6 ++++++
 arch/riscv/kvm/mmu.c    | 24 +++++++++++++++++++++++-
 2 files changed, 29 insertions(+), 1 deletion(-)
arch/riscv/kvm/gstage.c +6 -0
index e5002cb9cbef..54d45addf18f 100644
@@ -174,6 +174,12 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,
 
 	if (pte_val(*ptep) != pte_val(map->pte)) {
 		bool was_invalid = !pte_val(*ptep);
+
+		/* Avoid replacing an existing lower-level table with a leaf mapping. */
+		if (!gstage_pte_leaf(ptep) && !was_invalid &&
+		    gstage_pte_leaf(&map->pte))
+			return -EEXIST;
+
 		set_pte(ptep, map->pte);
 		if (gstage_pte_leaf(ptep) &&
 		    !(was_invalid && riscv_has_extension_unlikely(RISCV_ISA_EXT_SVVPTC)))
arch/riscv/kvm/mmu.c +23 -1
index 6035b5ec9503..bfd6168ebe30 100644
@@ -625,10 +625,11 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
 		      struct kvm_gstage_mapping *out_map)
 {
 	int ret;
-	kvm_pfn_t hfn;
+	kvm_pfn_t fault_hfn, hfn;
 	bool is_hugetlb;
 	bool writable;
 	unsigned int vma_pageshift;
+	gpa_t fault_gpa = gpa;
 	gfn_t gfn = gpa >> PAGE_SHIFT;
 	struct vm_area_struct *vma;
 	struct kvm *kvm = vcpu->kvm;
@@ -709,6 +710,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
 	}
 	if (is_error_noslot_pfn(hfn))
 		return -EFAULT;
+	fault_hfn = hfn + ((fault_gpa >> PAGE_SHIFT) - gfn);
 
 	/*
 	 * If logging is active then we allow writable pages only
@@ -734,9 +736,29 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
 		mark_page_dirty_in_slot(kvm, memslot, gfn);
 		ret = kvm_riscv_gstage_map_page(&gstage, pcache, gpa, hfn << PAGE_SHIFT,
 						vma_pagesize, false, true, out_map);
+		if (ret == -EEXIST) {
+			/*
+			 * Retry at 4K granularity for the original faulting GPA
+			 * when a huge leaf cannot replace an existing table.
+			 */
+			ret = kvm_riscv_gstage_map_page(&gstage, pcache, fault_gpa,
+							fault_hfn << PAGE_SHIFT,
+							PAGE_SIZE, false, true,
+							out_map);
+		}
 	} else {
 		ret = kvm_riscv_gstage_map_page(&gstage, pcache, gpa, hfn << PAGE_SHIFT,
 						vma_pagesize, true, true, out_map);
+		if (ret == -EEXIST) {
+			/*
+			 * Retry at 4K granularity for the original faulting GPA
+			 * when a huge leaf cannot replace an existing table.
+			 */
+			ret = kvm_riscv_gstage_map_page(&gstage, pcache, fault_gpa,
+							fault_hfn << PAGE_SHIFT,
+							PAGE_SIZE, true, true,
+							out_map);
+		}
 	}
 
 	if (ret)
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH v4 2/5 KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves

In reply to Jinyu Tang

When dirty logging is disabled after a G-stage PMD mapping has been split, the fault path may see a 4K G-stage leaf while the backing host page is still THP-backed. The existing kvm_riscv_gstage_map_page() comment says that this path should update the small leaf and leave huge mapping recovery to a later ioctl path.

However, transparent_hugepage_adjust() runs before that G-stage lookup and rewrites the fault GPA to the PMD base. If the original fault is not at the PMD base, the lookup can find and update the wrong 4K leaf.

Check the original fault GPA in transparent_hugepage_adjust(). If it already has a 4K G-stage leaf, skip THP adjustment and keep handling the fault at PAGE_SIZE granularity.

---
 arch/riscv/kvm/mmu.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
arch/riscv/kvm/mmu.c +13 -1
index bfd6168ebe30..2fabcd409991 100644
@@ -500,10 +500,21 @@ static int get_hva_mapping_size(struct kvm *kvm,
 
 static unsigned long transparent_hugepage_adjust(struct kvm *kvm,
 						 struct kvm_memory_slot *memslot,
+						 struct kvm_gstage *gstage,
 						 unsigned long hva,
 						 kvm_pfn_t *hfnp, gpa_t *gpa)
 {
 	kvm_pfn_t hfn = *hfnp;
+	u32 ptep_level;
+	pte_t *ptep;
+
+	/*
+	 * Keep the existing split G-stage leaf and update the original
+	 * faulting 4K page in the vCPU fault path.
+	 */
+	if (kvm_riscv_gstage_get_leaf(gstage, *gpa, &ptep, &ptep_level) &&
+	    !ptep_level)
+		return PAGE_SIZE;
 
 	/*
 	 * Make sure the adjustment is done only for THP pages. Also make
@@ -730,7 +741,8 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
 	 * so do not promote them through the THP helper.
 	 */
 	if (!logging && !is_hugetlb && vma_pagesize == PAGE_SIZE)
-		vma_pagesize = transparent_hugepage_adjust(kvm, memslot, hva, &hfn, &gpa);
+		vma_pagesize = transparent_hugepage_adjust(kvm, memslot, &gstage,
+							   hva, &hfn, &gpa);
 
 	if (writable) {
 		mark_page_dirty_in_slot(kvm, memslot, gfn);
Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH v4 2/5] KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Incomplete bug fix allows an infinite page fault loop when handling THP-backed memory. --

--- Patch [2]: [PATCH v4 2/5] KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves --- commit 6582fbd6641ea5762d156dac1847e91f07decbd7
Author: Jinyu Tang <[email protected]>

KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves

This commit updates transparent_hugepage_adjust() to check if the original fault GPA already has a 4K G-stage leaf. If so, it skips THP adjustment and handles the fault at PAGE_SIZE granularity.

Quoted 4 lines
arch/riscv/kvm/mmu.c +0 -0
index bfd6168ebe308..2fabcd4099919 100644

[ ... ]

Quoted 9 lines
arch/riscv/kvm/mmu.c +1 -0
@@ -500,10 +500,21 @@ static int get_hva_mapping_size(struct kvm *kvm,
 
 static unsigned long transparent_hugepage_adjust(struct kvm *kvm,
 						 struct kvm_memory_slot *memslot,
+						 struct kvm_gstage *gstage,
 						 unsigned long hva,
 						 kvm_pfn_t *hfnp, gpa_t *gpa)
 {
 	kvm_pfn_t hfn = *hfnp;

arch/riscv/kvm/mmu.c static int get_hva_mapping_size(struct kvm *kvm, · line 517

Quoted 10 lines
arch/riscv/kvm/mmu.c +10 -0
+	u32 ptep_level;
+	pte_t *ptep;
+
+	/*
+	 * Keep the existing split G-stage leaf and update the original
+	 * faulting 4K page in the vCPU fault path.
+	 */
+	if (kvm_riscv_gstage_get_leaf(gstage, *gpa, &ptep, &ptep_level) &&
+	    !ptep_level)
+		return PAGE_SIZE;

[Severity: High]
This is a pre-existing issue, but does this fully prevent the infinite page fault loop when handling THP-backed memory?

This check only validates the unaligned faulting GPA. If the faulting GPA is unmapped, it proceeds to align the GPA to the PMD base. However, if the PMD base itself already has a 4K leaf (e.g., from a prior access while dirty logging was enabled), transparent_hugepage_adjust() will still return PMD_SIZE.

When called with the aligned GPA (the PMD base), kvm_riscv_gstage_map_page() will find the 4K leaf at the PMD base:

arch/riscv/kvm/gstage.c:kvm_riscv_gstage_map_page() {
    ...
		} else if (ALIGN_DOWN(PFN_PHYS(pte_pfn(ptep_get(ptep))), page_size) == hpa) {
			kvm_riscv_gstage_update_pte_prot(gstage, ptep_level, gpa, ptep, prot);
			return 0;
		}
    ...
}

It updates the protection bits of the PMD base leaf and returns success without mapping the original faulting GPA. Could this leave the originally faulting GPA unmapped, causing the vCPU to hang in an infinite page fault loop?

Quoted 3 lines
arch/riscv/kvm/mmu.c +0 -0
 	/*
 	 * Make sure the adjustment is done only for THP pages. Also make
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
author
#4 lore raw

PATCH v4 3/5 KVM: riscv: Implement KVM_PRE_FAULT_MEMORY

In reply to Jinyu Tang

The generic KVM_PRE_FAULT_MEMORY ioctl lets userspace populate KVM page tables before running a vCPU over a GPA range. x86 already supports the ioctl, but RISC-V does not expose the capability and has no arch hook.

Add the RISC-V arch hook and reuse the existing G-stage fault mapping path with a read access. Report progress using the G-stage mapping returned by the map path, so the ioctl can advance by the actual leaf size that covers the requested GPA. Retry until a mapping is installed or a signal, VM-dead request, or real error is observed.

---
 arch/riscv/kvm/Kconfig  |  1 +
 arch/riscv/kvm/gstage.c |  3 +++
 arch/riscv/kvm/mmu.c    | 45 ++++++++++++++++++++++++++++++++++++++++-
 arch/riscv/kvm/vm.c     |  1 +
 4 files changed, 49 insertions(+), 1 deletion(-)
arch/riscv/kvm/Kconfig +1 -0
index ec2cee0a39e0..8ac209e8ac87 100644
@@ -28,6 +28,7 @@ config KVM
 	select KVM_COMMON
 	select KVM_GENERIC_DIRTYLOG_READ_PROTECT
 	select KVM_GENERIC_HARDWARE_ENABLING
+	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select KVM_MMIO
 	select VIRT_XFER_TO_GUEST_WORK
 	select SCHED_INFO
arch/riscv/kvm/gstage.c +3 -0
index 54d45addf18f..dff315dfd24e 100644
@@ -286,6 +286,9 @@ int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage,
 						    out_map->level, true);
 		} else if (ALIGN_DOWN(PFN_PHYS(pte_pfn(ptep_get(ptep))), page_size) == hpa) {
 			kvm_riscv_gstage_update_pte_prot(gstage, ptep_level, gpa, ptep, prot);
+			out_map->addr = ALIGN_DOWN(gpa, page_size);
+			out_map->level = ptep_level;
+			out_map->pte = ptep_get(ptep);
 			return 0;
 		}
 	}
arch/riscv/kvm/mmu.c +44 -1

57 lines not on this page · raw message

arch/riscv/kvm/vm.c +1 -0
index a9f083feeb76..58500a19b33b 100644
@@ -187,6 +187,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_MP_STATE:
 	case KVM_CAP_IMMEDIATE_EXIT:
 	case KVM_CAP_SET_GUEST_DEBUG:
+	case KVM_CAP_PRE_FAULT_MEMORY:
 		r = 1;
 		break;
 	case KVM_CAP_NR_VCPUS:
Signature · 1 line
2.43.0
#5 lore raw

Re: [PATCH v4 3/5] KVM: riscv: Implement KVM_PRE_FAULT_MEMORY

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Returning `-EFAULT` on `KVM_PFN_ERR_HWPOISON` in `kvm_riscv_mmu_map` breaks `KVM_RUN` execution…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Returning -EFAULT on KVM_PFN_ERR_HWPOISON in kvm_riscv_mmu_map breaks KVM_RUN execution and normal KVM signal delivery.
- [High] Incorrect map advancement calculation in kvm_arch_vcpu_pre_fault_memory causes skipped pages and false completion for unaligned and split huge pages. --

--- Patch [3]: [PATCH v4 3/5] KVM: riscv: Implement KVM_PRE_FAULT_MEMORY --- commit d1fe5f152fd69a51f4bc7397e89186eff774f433
Author: Jinyu Tang <[email protected]>
Subject: KVM: riscv: Implement KVM_PRE_FAULT_MEMORY

The patch implements the KVM_PRE_FAULT_MEMORY ioctl for RISC-V by adding the kvm_arch_vcpu_pre_fault_memory() hook. It reuses the existing G-stage fault mapping path to populate KVM page tables.

Quoted 14 lines
arch/riscv/kvm/gstage.c +3 -0
index 54d45addf18f9..dff315dfd24ed 100644
@@ -286,6 +286,9 @@ int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage,
 						    out_map->level, true);
 		} else if (ALIGN_DOWN(PFN_PHYS(pte_pfn(ptep_get(ptep))), page_size) == hpa) {
 			kvm_riscv_gstage_update_pte_prot(gstage, ptep_level, gpa, ptep, prot);
+			out_map->addr = ALIGN_DOWN(gpa, page_size);
+			out_map->level = ptep_level;
+			out_map->pte = ptep_get(ptep);
 			return 0;
 		}
 	}

[ ... ]

arch/riscv/kvm/mmu.c int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, · line 720

Quoted 10 lines
arch/riscv/kvm/mmu.c +1 -1
index 2fabcd4099919..f6ca86da53eb8 100644
@@ -717,7 +717,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
 	if (hfn == KVM_PFN_ERR_HWPOISON) {
 		send_sig_mceerr(BUS_MCEERR_AR, (void __user *)hva,
 				vma_pageshift, current);
-		return 0;
+		return -EFAULT;

[Severity: High]
Does returning -EFAULT here break KVM's standard exit-to-userspace flow on a signal?

If a guest accesses a poisoned memory page, returning 0 allows KVM to bubble up 1 to kvm_arch_vcpu_ioctl_run(), which checks signal_pending() (set by send_sig_mceerr) and correctly exits to userspace with -EINTR.

Returning -EFAULT causes the KVM_RUN ioctl to immediately fail with -EFAULT, which can cause the VMM to crash instead of gracefully handling the Machine Check Exception.

[ ... ]

Quoted 1 line
arch/riscv/kvm/mmu.c +0 -0
@@ -782,6 +782,49 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,

[ ... ]

Quoted 12 lines
arch/riscv/kvm/mmu.c +12 -0
+	for (;;) {
+		if (signal_pending(current))
+			return -EINTR;
+
+		if (kvm_check_request(KVM_REQ_VM_DEAD, vcpu))
+			return -EIO;
+
+		cond_resched();
+		ret = kvm_riscv_mmu_map(vcpu, memslot, range->gpa, hva, false, &out_map);
+		if (ret)
+			return ret;
+

arch/riscv/kvm/mmu.c int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, · line 826

Quoted 10 lines
arch/riscv/kvm/mmu.c +10 -0
+		if (!pte_val(out_map.pte))
+			continue;
+
+		map_size = PAGE_SIZE << (out_map.level * kvm_riscv_gstage_index_bits);
+		end = out_map.addr + map_size;
+		break;
+	}
+
+	return min_t(u64, range->size, end - range->gpa);
+}

[Severity: High]
Can end - range->gpa underflow here when handling unaligned offsets or split huge pages?

In kvm_riscv_gstage_map_page() above, out_map.addr is aligned to the requested huge page size rather than the actual leaf size installed.

If out_map.level is set to the 4K leaf level, but out_map.addr is aligned to the huge page size, and range->gpa is beyond the first 4K of that huge page, end will be calculated as less than range->gpa.

This would cause end - range->gpa to underflow into a large u64, making min_t() evaluate to range->size and prematurely terminating the ioctl while skipping regions of guest physical address space.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author
#6 lore raw

PATCH v4 4/5 KVM: selftests: Add RISC-V Sv57 page table indexing

In reply to Jinyu Tang

RISC-V selftests can create guests with five page-table levels, for example when the selected guest mode is Sv57. The RISC-V page-table walker only had index arrays for levels 0 through 3, so virt_arch_pg_map() indexed past the end of the arrays when level 4 was used.

Add the missing L4 index mask and shift so selftests can build guest page tables for Sv57 VMs.

---
 tools/testing/selftests/kvm/include/riscv/processor.h | 3 +++
 tools/testing/selftests/kvm/lib/riscv/processor.c     | 2 ++
 2 files changed, 5 insertions(+)
tools/testing/selftests/kvm/include/riscv/processor.h +3 -0
index e3acf2ae9881..abde3c71c891 100644
@@ -127,6 +127,9 @@ void vm_install_exception_handler(struct kvm_vm *vm, int vector, exception_handl
 
 void vm_install_interrupt_handler(struct kvm_vm *vm, exception_handler_fn handler);
 
+/* L4 index Bit[56:48] */
+#define PGTBL_L4_INDEX_MASK			0x01FF000000000000ULL
+#define PGTBL_L4_INDEX_SHIFT			48
 /* L3 index Bit[47:39] */
 #define PGTBL_L3_INDEX_MASK			0x0000FF8000000000ULL
 #define PGTBL_L3_INDEX_SHIFT			39
tools/testing/selftests/kvm/lib/riscv/processor.c +2 -0
index ded5429f3448..b4d41a407553 100644
@@ -43,6 +43,7 @@ static u64 pte_index_mask[] = {
 	PGTBL_L1_INDEX_MASK,
 	PGTBL_L2_INDEX_MASK,
 	PGTBL_L3_INDEX_MASK,
+	PGTBL_L4_INDEX_MASK,
 };
 
 static u32 pte_index_shift[] = {
@@ -50,6 +51,7 @@ static u32 pte_index_shift[] = {
 	PGTBL_L1_INDEX_SHIFT,
 	PGTBL_L2_INDEX_SHIFT,
 	PGTBL_L3_INDEX_SHIFT,
+	PGTBL_L4_INDEX_SHIFT,
 };
 
 static u64 pte_index(struct kvm_vm *vm, gva_t gva, int level)
Signature · 1 line
2.43.0
author
#7 lore raw

PATCH v4 5/5 KVM: selftests: Enable pre_fault_memory_test for RISC-V

In reply to Jinyu Tang

RISC-V now supports KVM_PRE_FAULT_MEMORY, so include the generic pre_fault_memory_test in the RISC-V KVM selftest build.

The test uses PAGE_SIZE from the architecture processor header. Define the normal 4K RISC-V selftest page size so the generic test can build for RISC-V.

---
 tools/testing/selftests/kvm/Makefile.kvm              | 1 +
 tools/testing/selftests/kvm/include/riscv/processor.h | 3 +++
 2 files changed, 4 insertions(+)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 6fc34e9bf8e1..ac64ac92fd4b 100644
@@ -225,6 +225,7 @@ TEST_GEN_PROGS_riscv += coalesced_io_test
 TEST_GEN_PROGS_riscv += dirty_log_perf_test
 TEST_GEN_PROGS_riscv += get-reg-list
 TEST_GEN_PROGS_riscv += mmu_stress_test
+TEST_GEN_PROGS_riscv += pre_fault_memory_test
 TEST_GEN_PROGS_riscv += rseq_test
 TEST_GEN_PROGS_riscv += steal_time
 
tools/testing/selftests/kvm/include/riscv/processor.h +3 -0
index abde3c71c891..70487c8ed155 100644
@@ -12,6 +12,9 @@
 #include <asm/vdso/processor.h>
 #include "kvm_util.h"
 
+#define PAGE_SHIFT		12
+#define PAGE_SIZE		BIT_ULL(PAGE_SHIFT)
+
 #define INSN_OPCODE_MASK	0x007c
 #define INSN_OPCODE_SHIFT	2
 #define INSN_OPCODE_SYSTEM	28
Signature · 1 line
2.43.0