KVM: riscv: Add KVM_PRE_FAULT_MEMORY support
Subsystems (4)
v4 cover KVM: riscv: Add KVM_PRE_FAULT_MEMORY support
Add RISC-V support for the generic KVM_PRE_FAULT_MEMORY ioctl.
Patch 1 and patch 2 fix existing G-stage huge-page corner cases. They are included first because the prefault hook relies on the map path reporting the mapping that actually covers the requested GPA.
Patch 3 adds the RISC-V arch hook for KVM_PRE_FAULT_MEMORY. Patch 4 adds Sv57 indexing support to the RISC-V KVM selftest helpers, and patch 5 enables the generic pre_fault_memory_test for RISC-V.
Patch 1 folds the previously posted standalone G-stage table overwrite fix into this series:
https://lore.kernel.org/kvm-riscv/[email protected]/T/#u
Changes in v4:
- Fold the standalone G-stage non-leaf overwrite fix into this series.
- Avoid THP adjustment when the original faulting GPA already has an existing 4K G-stage leaf.
- Return an error for HWPOISON instead of reporting success without a visible G-stage mapping. (Sashiko)
Changes in v3:
- Retry internally when the map path returns success without a visible G-stage mapping, instead of exposing -EAGAIN. (Sashiko)
Changes in v2:
- Drop the per-test guest_modes_append_default() call from pre_fault_memory_test.c. (Sean)
Jinyu Tang (5):
KVM: riscv: Avoid overwriting G-stage tables with huge leaves
KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves
KVM: riscv: Implement KVM_PRE_FAULT_MEMORY
KVM: selftests: Add RISC-V Sv57 page table indexing
KVM: selftests: Enable pre_fault_memory_test for RISC-V
arch/riscv/kvm/Kconfig | 1 + arch/riscv/kvm/gstage.c | 9 ++ arch/riscv/kvm/mmu.c | 83 ++++++++++++++++++- arch/riscv/kvm/vm.c | 1 + tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/include/riscv/processor.h | 6 ++ .../selftests/kvm/lib/riscv/processor.c | 2 + 7 files changed, 100 insertions(+), 3 deletions(-)
Signature · 1 line
2.43.0
PATCH v4 1/5 KVM: riscv: Avoid overwriting G-stage tables with huge leaves
RISC-V KVM can overwrite an existing G-stage table entry when installing a huge leaf mapping. If the target huge range already has a lower-level page table, kvm_riscv_gstage_set_pte() can replace the non-leaf entry with a leaf PTE and disconnect the lower-level page table.
Reject replacing a valid table entry with a leaf PTE. If huge-page installation hits such a conflict, fall back to a 4K mapping for the original faulting GPA in the MMU fault path, where the original GPA and HFN are still available.
--- arch/riscv/kvm/gstage.c | 6 ++++++ arch/riscv/kvm/mmu.c | 24 +++++++++++++++++++++++- 2 files changed, 29 insertions(+), 1 deletion(-)
arch/riscv/kvm/gstage.c +6 -0
@@ -174,6 +174,12 @@ int kvm_riscv_gstage_set_pte(struct kvm_gstage *gstage,if (pte_val (*ptep) !=pte_val (map->pte)) {bool was_invalid = !pte_val (*ptep); + +/* Avoid replacing an existing lower-level table with a leaf mapping. */ +if (!gstage_pte_leaf (ptep) && !was_invalid && +gstage_pte_leaf (&map->pte)) +return -EEXIST ; +set_pte (ptep, map->pte);if (gstage_pte_leaf (ptep) && !(was_invalid &&riscv_has_extension_unlikely (RISCV_ISA_EXT_SVVPTC )))
arch/riscv/kvm/mmu.c +23 -1
@@ -625,10 +625,11 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, struct kvm_gstage_mapping *out_map) {int ret; -kvm_pfn_t hfn; +kvm_pfn_t fault_hfn, hfn;bool is_hugetlb;bool writable;unsigned int vma_pageshift; +gpa_t fault_gpa = gpa;gfn_t gfn = gpa >>PAGE_SHIFT ;struct vm_area_struct *vma;struct kvm *kvm = vcpu->kvm;
@@ -709,6 +710,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, }if (is_error_noslot_pfn (hfn))return -EFAULT ; + fault_hfn = hfn + ((fault_gpa >>PAGE_SHIFT ) - gfn);/* * If logging is active then we allow writable pages only
@@ -734,9 +736,29 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,mark_page_dirty_in_slot (kvm, memslot, gfn); ret =kvm_riscv_gstage_map_page (&gstage, pcache, gpa, hfn <<PAGE_SHIFT , vma_pagesize, false, true, out_map); +if (ret == -EEXIST ) { +/* +* Retry at 4K granularity for the original faulting GPA +* when a huge leaf cannot replace an existing table. +*/ + ret =kvm_riscv_gstage_map_page (&gstage, pcache, fault_gpa, + fault_hfn <<PAGE_SHIFT , +PAGE_SIZE , false, true, + out_map); + } }else { ret =kvm_riscv_gstage_map_page (&gstage, pcache, gpa, hfn <<PAGE_SHIFT , vma_pagesize, true, true, out_map); +if (ret == -EEXIST ) { +/* +* Retry at 4K granularity for the original faulting GPA +* when a huge leaf cannot replace an existing table. +*/ + ret =kvm_riscv_gstage_map_page (&gstage, pcache, fault_gpa, + fault_hfn <<PAGE_SHIFT , +PAGE_SIZE , true, true, + out_map); + } }if (ret)
Signature · 1 line
2.43.0
PATCH v4 2/5 KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves
In reply to Jinyu TangWhen dirty logging is disabled after a G-stage PMD mapping has been split, the fault path may see a 4K G-stage leaf while the backing host page is still THP-backed. The existing kvm_riscv_gstage_map_page() comment says that this path should update the small leaf and leave huge mapping recovery to a later ioctl path.
However, transparent_hugepage_adjust() runs before that G-stage lookup and rewrites the fault GPA to the PMD base. If the original fault is not at the PMD base, the lookup can find and update the wrong 4K leaf.
Check the original fault GPA in transparent_hugepage_adjust(). If it already has a 4K G-stage leaf, skip THP adjustment and keep handling the fault at PAGE_SIZE granularity.
--- arch/riscv/kvm/mmu.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
arch/riscv/kvm/mmu.c +13 -1
@@ -500,10 +500,21 @@ static int get_hva_mapping_size(struct kvm *kvm,static unsigned long transparent_hugepage_adjust (struct kvm *kvm,struct kvm_memory_slot *memslot, +struct kvm_gstage *gstage,unsigned long hva,kvm_pfn_t *hfnp,gpa_t *gpa) {kvm_pfn_t hfn = *hfnp; +u32 ptep_level; +pte_t *ptep; + +/* +* Keep the existing split G-stage leaf and update the original +* faulting 4K page in the vCPU fault path. +*/ +if (kvm_riscv_gstage_get_leaf (gstage, *gpa, &ptep, &ptep_level) && + !ptep_level) +return PAGE_SIZE ;/* * Make sure the adjustment is done only for THP pages. Also make
@@ -730,7 +741,8 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,* so do not promote them through the THP helper. */ if (!logging && !is_hugetlb && vma_pagesize ==PAGE_SIZE ) - vma_pagesize =transparent_hugepage_adjust (kvm, memslot, hva, &hfn, &gpa); + vma_pagesize =transparent_hugepage_adjust (kvm, memslot, &gstage, + hva, &hfn, &gpa);if (writable) {mark_page_dirty_in_slot (kvm, memslot, gfn);
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v4 2/5] KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Incomplete bug fix allows an infinite page fault loop when handling THP-backed memory. --
--- Patch [2]: [PATCH v4 2/5] KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves --- commit 6582fbd6641ea5762d156dac1847e91f07decbd7
Author: Jinyu Tang <[email protected]>
KVM: riscv: Avoid THP adjustment for existing 4K G-stage leaves
This commit updates transparent_hugepage_adjust() to check if the original fault GPA already has a 4K G-stage leaf. If so, it skips THP adjustment and handles the fault at PAGE_SIZE granularity.
Quoted 4 lines
arch/riscv/kvm/mmu.c +0
-0
[ ... ]
Quoted 9 lines
arch/riscv/kvm/mmu.c +1
-0@@ -500,10 +500,21 @@ static int get_hva_mapping_size(struct kvm *kvm,static unsigned long transparent_hugepage_adjust (struct kvm *kvm,struct kvm_memory_slot *memslot, +struct kvm_gstage *gstage,unsigned long hva,kvm_pfn_t *hfnp,gpa_t *gpa) {kvm_pfn_t hfn = *hfnp;
arch/riscv/kvm/mmu.c static int get_hva_mapping_size(struct kvm *kvm, · line 517
Quoted 10 lines
arch/riscv/kvm/mmu.c +10
-0+u32 ptep_level; +pte_t *ptep; + +/* +* Keep the existing split G-stage leaf and update the original +* faulting 4K page in the vCPU fault path. +*/ +if (kvm_riscv_gstage_get_leaf (gstage, *gpa, &ptep, &ptep_level) && + !ptep_level) +return PAGE_SIZE ;
[Severity: High]
This is a pre-existing issue, but does this fully prevent the infinite page fault loop when handling THP-backed memory?
This check only validates the unaligned faulting GPA. If the faulting GPA is unmapped, it proceeds to align the GPA to the PMD base. However, if the PMD base itself already has a 4K leaf (e.g., from a prior access while dirty logging was enabled), transparent_hugepage_adjust() will still return PMD_SIZE.
When called with the aligned GPA (the PMD base), kvm_riscv_gstage_map_page() will find the 4K leaf at the PMD base:
arch/riscv/kvm/gstage.c:kvm_riscv_gstage_map_page() {
...
} else if (ALIGN_DOWN(PFN_PHYS(pte_pfn(ptep_get(ptep))), page_size) == hpa) {
kvm_riscv_gstage_update_pte_prot(gstage, ptep_level, gpa, ptep, prot);
return 0;
}
...
}It updates the protection bits of the PMD base leaf and returns success without mapping the original faulting GPA. Could this leave the originally faulting GPA unmapped, causing the vCPU to hang in an infinite page fault loop?
Quoted 3 lines
arch/riscv/kvm/mmu.c +0
-0/* * Make sure the adjustment is done only for THP pages. Also make
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
PATCH v4 3/5 KVM: riscv: Implement KVM_PRE_FAULT_MEMORY
In reply to Jinyu TangThe generic KVM_PRE_FAULT_MEMORY ioctl lets userspace populate KVM page tables before running a vCPU over a GPA range. x86 already supports the ioctl, but RISC-V does not expose the capability and has no arch hook.
Add the RISC-V arch hook and reuse the existing G-stage fault mapping path with a read access. Report progress using the G-stage mapping returned by the map path, so the ioctl can advance by the actual leaf size that covers the requested GPA. Retry until a mapping is installed or a signal, VM-dead request, or real error is observed.
--- arch/riscv/kvm/Kconfig | 1 + arch/riscv/kvm/gstage.c | 3 +++ arch/riscv/kvm/mmu.c | 45 ++++++++++++++++++++++++++++++++++++++++- arch/riscv/kvm/vm.c | 1 + 4 files changed, 49 insertions(+), 1 deletion(-)
arch/riscv/kvm/Kconfig +1 -0
@@ -28,6 +28,7 @@ config KVM select KVM_COMMON select KVM_GENERIC_DIRTYLOG_READ_PROTECT select KVM_GENERIC_HARDWARE_ENABLING + select KVM_GENERIC_PRE_FAULT_MEMORY select KVM_MMIO select VIRT_XFER_TO_GUEST_WORK select SCHED_INFO
arch/riscv/kvm/gstage.c +3 -0
@@ -286,6 +286,9 @@ int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, out_map->level, true); }else if (ALIGN_DOWN (PFN_PHYS (pte_pfn (ptep_get (ptep))), page_size) == hpa) {kvm_riscv_gstage_update_pte_prot (gstage, ptep_level, gpa, ptep, prot); + out_map->addr =ALIGN_DOWN (gpa, page_size); + out_map->level = ptep_level; + out_map->pte =ptep_get (ptep);return 0 ; } }
arch/riscv/kvm/mmu.c +44 -1
57 lines not on this page · raw message
arch/riscv/kvm/vm.c +1 -0
@@ -187,6 +187,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_MP_STATE :case KVM_CAP_IMMEDIATE_EXIT :case KVM_CAP_SET_GUEST_DEBUG : +case KVM_CAP_PRE_FAULT_MEMORY : r =1 ;break ;case KVM_CAP_NR_VCPUS :
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v4 3/5] KVM: riscv: Implement KVM_PRE_FAULT_MEMORY
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] Returning `-EFAULT` on `KVM_PFN_ERR_HWPOISON` in `kvm_riscv_mmu_map` breaks `KVM_RUN` execution…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Returning -EFAULT on KVM_PFN_ERR_HWPOISON in kvm_riscv_mmu_map breaks KVM_RUN execution and normal KVM signal delivery.
- [High] Incorrect map advancement calculation in kvm_arch_vcpu_pre_fault_memory causes skipped pages and false completion for unaligned and split huge pages. --
--- Patch [3]: [PATCH v4 3/5] KVM: riscv: Implement KVM_PRE_FAULT_MEMORY --- commit d1fe5f152fd69a51f4bc7397e89186eff774f433
Author: Jinyu Tang <[email protected]>
Subject: KVM: riscv: Implement KVM_PRE_FAULT_MEMORY
The patch implements the KVM_PRE_FAULT_MEMORY ioctl for RISC-V by adding the kvm_arch_vcpu_pre_fault_memory() hook. It reuses the existing G-stage fault mapping path to populate KVM page tables.
Quoted 14 lines
arch/riscv/kvm/gstage.c +3
-0@@ -286,6 +286,9 @@ int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, out_map->level, true); }else if (ALIGN_DOWN (PFN_PHYS (pte_pfn (ptep_get (ptep))), page_size) == hpa) {kvm_riscv_gstage_update_pte_prot (gstage, ptep_level, gpa, ptep, prot); + out_map->addr =ALIGN_DOWN (gpa, page_size); + out_map->level = ptep_level; + out_map->pte =ptep_get (ptep);return 0 ; } }
[ ... ]
arch/riscv/kvm/mmu.c int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, · line 720
Quoted 10 lines
arch/riscv/kvm/mmu.c +1
-1@@ -717,7 +717,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,if (hfn ==KVM_PFN_ERR_HWPOISON ) {send_sig_mceerr (BUS_MCEERR_AR , (void __user *)hva, vma_pageshift, current); -return 0 ; +return -EFAULT ;
[Severity: High]
Does returning -EFAULT here break KVM's standard exit-to-userspace flow on a signal?
If a guest accesses a poisoned memory page, returning 0 allows KVM to bubble up 1 to kvm_arch_vcpu_ioctl_run(), which checks signal_pending() (set by send_sig_mceerr) and correctly exits to userspace with -EINTR.
Returning -EFAULT causes the KVM_RUN ioctl to immediately fail with -EFAULT, which can cause the VMM to crash instead of gracefully handling the Machine Check Exception.
[ ... ]
Quoted 1 line
arch/riscv/kvm/mmu.c +0
-0@@ -782,6 +782,49 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
[ ... ]
Quoted 12 lines
arch/riscv/kvm/mmu.c +12
-0+for (;;) { +if (signal_pending (current)) +return -EINTR ; + +if (kvm_check_request (KVM_REQ_VM_DEAD , vcpu)) +return -EIO ; + +cond_resched (); + ret =kvm_riscv_mmu_map (vcpu, memslot, range->gpa, hva, false, &out_map); +if (ret) +return ret; +
arch/riscv/kvm/mmu.c int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, · line 826
Quoted 10 lines
arch/riscv/kvm/mmu.c +10
-0+if (!pte_val (out_map.pte)) +continue ; + + map_size =PAGE_SIZE << (out_map.level * kvm_riscv_gstage_index_bits); + end = out_map.addr + map_size; +break ; + } + +return min_t (u64, range->size, end - range->gpa); +}
[Severity: High]
Can end - range->gpa underflow here when handling unaligned offsets or split huge pages?
In kvm_riscv_gstage_map_page() above, out_map.addr is aligned to the requested huge page size rather than the actual leaf size installed.
If out_map.level is set to the 4K leaf level, but out_map.addr is aligned to the huge page size, and range->gpa is beyond the first 4K of that huge page, end will be calculated as less than range->gpa.
This would cause end - range->gpa to underflow into a large u64, making min_t() evaluate to range->size and prematurely terminating the ioctl while skipping regions of guest physical address space.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v4 4/5 KVM: selftests: Add RISC-V Sv57 page table indexing
In reply to Jinyu TangRISC-V selftests can create guests with five page-table levels, for example when the selected guest mode is Sv57. The RISC-V page-table walker only had index arrays for levels 0 through 3, so virt_arch_pg_map() indexed past the end of the arrays when level 4 was used.
Add the missing L4 index mask and shift so selftests can build guest page tables for Sv57 VMs.
--- tools/testing/selftests/kvm/include/riscv/processor.h | 3 +++ tools/testing/selftests/kvm/lib/riscv/processor.c | 2 ++ 2 files changed, 5 insertions(+)
tools/testing/selftests/kvm/include/riscv/processor.h +3 -0
@@ -127,6 +127,9 @@ void vm_install_exception_handler(struct kvm_vm *vm, int vector, exception_handlvoid vm_install_interrupt_handler (struct kvm_vm *vm,exception_handler_fn handler); +/* L4 index Bit[56:48] */ +#define PGTBL_L4_INDEX_MASK 0x01FF000000000000ULL +#define PGTBL_L4_INDEX_SHIFT 48/* L3 index Bit[47:39] */ #define PGTBL_L3_INDEX_MASK 0x0000FF8000000000ULL#define PGTBL_L3_INDEX_SHIFT 39
tools/testing/selftests/kvm/lib/riscv/processor.c +2 -0
@@ -43,6 +43,7 @@ static u64 pte_index_mask[] = {PGTBL_L1_INDEX_MASK ,PGTBL_L2_INDEX_MASK ,PGTBL_L3_INDEX_MASK , + PGTBL_L4_INDEX_MASK, }; static u32 pte_index_shift[] = {
@@ -50,6 +51,7 @@ static u32 pte_index_shift[] = {PGTBL_L1_INDEX_SHIFT ,PGTBL_L2_INDEX_SHIFT ,PGTBL_L3_INDEX_SHIFT , + PGTBL_L4_INDEX_SHIFT, };static u64 pte_index (struct kvm_vm *vm,gva_t gva,int level)
Signature · 1 line
2.43.0
PATCH v4 5/5 KVM: selftests: Enable pre_fault_memory_test for RISC-V
In reply to Jinyu TangRISC-V now supports KVM_PRE_FAULT_MEMORY, so include the generic pre_fault_memory_test in the RISC-V KVM selftest build.
The test uses PAGE_SIZE from the architecture processor header. Define the normal 4K RISC-V selftest page size so the generic test can build for RISC-V.
--- tools/testing/selftests/kvm/Makefile.kvm | 1 + tools/testing/selftests/kvm/include/riscv/processor.h | 3 +++ 2 files changed, 4 insertions(+)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -225,6 +225,7 @@ TEST_GEN_PROGS_riscv += coalesced_io_test TEST_GEN_PROGS_riscv += dirty_log_perf_test TEST_GEN_PROGS_riscv += get-reg-list TEST_GEN_PROGS_riscv += mmu_stress_test +TEST_GEN_PROGS_riscv += pre_fault_memory_test TEST_GEN_PROGS_riscv += rseq_test TEST_GEN_PROGS_riscv += steal_time
tools/testing/selftests/kvm/include/riscv/processor.h +3 -0
@@ -12,6 +12,9 @@#include <asm/vdso/processor.h> #include "kvm_util.h" +#define PAGE_SHIFT 12 +#define PAGE_SIZE BIT_ULL(PAGE_SHIFT) +#define INSN_OPCODE_MASK 0x007c#define INSN_OPCODE_SHIFT 2#define INSN_OPCODE_SYSTEM 28
Signature · 1 line
2.43.0