KVM: Bound KVM_PRE_FAULT_MEMORY by the vCPU's dirty ring
Subsystems (7)
Fuad Tabba
reviewerv2 cover KVM: Bound KVM_PRE_FAULT_MEMORY by the vCPU's dirty ring
Hi folks,
Changes since v1 [1]:
- Switched the check to kvm_dirty_ring_check_request(). (Sean)
- The test checks that each page is reported exactly once, whatever the order. (Lorenzo)
- Fixed the test on 64K-page arm64 hosts by sizing its range in host pages.
- Various fixes. (Lorenzo)
- Dropped Lorenzo's Reviewed-by on patch 1, since the check changed.
KVM_PRE_FAULT_MEMORY never checks the vCPU's dirty ring, so one call over a large enough range of populated, dirty-logged pages overflows it. The first patch stops the loop at the soft limit, as KVM_RUN does. The second patch adds the case to pre_fault_memory_test.
Pre-faulting isn't the only ioctl that can push past the ring's reserve. On x86, KVM_SET_MSRS does too when it writes the wall clock or the Hyper-V hypercall page. A guest can also push past the reserve within one exit, for example through the arm64 ITS. What those paths should do when the ring fills is an open question for the dirty ring API, which I don't think belongs in this series.
I didn't include the kvm_dirty_ring_check_request() rewrite that Sean sketched on v1, since I think it can lose a request. It tests the request first and clears it afterwards. If the vCPU is scheduled out in between, loongarch sets the PV preempted flag in guest memory and x86 updates a Xen guest's runstate area, either of which pushes an entry. When that push makes the ring soft full, the request it makes is then cleared, and the vCPU runs with the ring soft full until its next push. Clearing first and re-setting, as the code does now, doesn't have that window.
Based on Linux 7.4-rc2 (7475ea8b58616; also applies to kvm-x86/next cbe54fc3462b3 with git am -3).
Cheers,
/fuad
[1] https://lore.kernel.org/all/[email protected]/
Fuad Tabba (2):
KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
KVM: selftests: Test KVM_PRE_FAULT_MEMORY against a full dirty ring
Documentation/virt/kvm/api.rst | 9 +- include/linux/kvm_dirty_ring.h | 5 + .../selftests/kvm/pre_fault_memory_test.c | 101 ++++++++++++++++++ virt/kvm/kvm_main.c | 5 + 4 files changed, 118 insertions(+), 2 deletions(-)
Signature · 1 line
2.47.3
Fuad Tabba
reviewerPATCH v2 1/2 KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
Pre-faulting can map a dirty-logged page writable, which pushes it to the vCPU's dirty ring. KVM_RUN exits with KVM_EXIT_DIRTY_RING_FULL before the ring overflows, but the pre-fault loop never checks it, so one call can push more entries than the ring has free, which trips the WARN in kvm_dirty_ring_push() and overwrites entries userspace hasn't harvested. On x86:
WARNING: arch/x86/kvm/../../../virt/kvm/dirty_ring.c:225 at kvm_dirty_ring_push+0x7a/0xd0 Call Trace: make_spte+0x2cc/0x450 kvm_tdp_mmu_map+0x5e1/0x7f0 kvm_tdp_page_fault+0x12c/0x1b0 kvm_mmu_do_page_fault+0x181/0x250 kvm_arch_vcpu_pre_fault_memory+0x136/0x1d0 kvm_vcpu_pre_fault_memory+0xcf/0x180 kvm_vcpu_ioctl+0x455/0x770
Stop the loop once the ring is soft full, with the check KVM_RUN makes before entering the guest. Return the pages already processed, or -ENOSPC when there aren't any, since a retry can't succeed until userspace harvests and resets the ring. s390 has the ioctl but no ring, so give kvm_dirty_ring_check_request() a stub.
The check also sets run->exit_reason, and arm64's KVM_RUN completes a pending MMIO exit only if exit_reason is still KVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the next KVM_RUN doesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit with KVM_RUN before it pre-faults.
--- Documentation/virt/kvm/api.rst | 9 +++++++-- include/linux/kvm_dirty_ring.h | 5 +++++ virt/kvm/kvm_main.c | 5 +++++ 3 files changed, 17 insertions(+), 2 deletions(-)
Documentation/virt/kvm/api.rst +7 -2
@@ -6544,6 +6544,7 @@ Errors: ENOENT The specified `gpa` is outside defined memslots. ENOEXEC The vCPU has not been initialised (arm64). EINTR An unmasked signal is pending and no page was processed. + ENOSPC The vCPU's dirty ring is full and no page was processed. EFAULT The parameter address was invalid. EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the
@@ -6588,7 +6589,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. +If that fails with ENOSPC, the vCPU's dirty ring has to be harvested +and reset first. On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address.
@@ -8957,7 +8960,9 @@ Therefore, the ioctl must be called *before* reading the content of the dirty pages. The dirty ring can get full. When it happens, the KVM_RUN of the -vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL. +vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and +KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the +ring has been harvested and reset. The dirty ring interface has a major difference comparing to the KVM_GET_DIRTY_LOG interface in that, when reading the dirty ring from
include/linux/kvm_dirty_ring.h +5 -0
@@ -60,6 +60,11 @@ static inline void kvm_dirty_ring_push(struct kvm_vcpu *vcpu, { } +static inline bool kvm_dirty_ring_check_request (struct kvm_vcpu *vcpu) +{ +return false; +} +static inline struct page *kvm_dirty_ring_get_page (struct kvm_dirty_ring *ring,u32 offset) {
virt/kvm/kvm_main.c +5 -0
@@ -4461,6 +4461,11 @@ static int kvm_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,break ; } +if (kvm_dirty_ring_check_request (vcpu)) { + r = -ENOSPC ; +break ; + } + r =kvm_arch_vcpu_pre_fault_memory (vcpu, range);if (WARN_ON_ONCE (r ==0 || r == -EIO ))break ;
Signature · 1 line
2.47.3
Fuad Tabba
reviewerPATCH v2 2/2 KVM: selftests: Test KVM_PRE_FAULT_MEMORY against a full dirty ring
In reply to Fuad TabbaPre-fault more populated, dirty-logged pages than the vCPU's dirty ring holds, and check that each call stops before the ring overflows, that a retry returns ENOSPC without progress, and that each page is reported exactly once. A read pre-fault can map a page read-only, or access-tracked on x86 without EPT A/D bits, so skip the case if pre-faulting doesn't dirty any page.
--- .../selftests/kvm/pre_fault_memory_test.c | 101 ++++++++++++++++++ 1 file changed, 101 insertions(+)
tools/testing/selftests/kvm/pre_fault_memory_test.c +101 -0
@@ -5,6 +5,9 @@* Author: * Isaku Yamahata <isaku.yamahata at gmail.com> */ +#include <asm/barrier.h> +#include <linux/bitmap.h> +#include <linux/bitops.h> #include <linux/sizes.h> #include <test_util.h>
@@ -17,6 +20,8 @@/* Arbitrarily chosen values */ #define TEST_BASE_SIZE SZ_2M#define TEST_SLOT 10 +/* Large enough for the entries every architecture reserves. */ +#define TEST_RING_SIZE SZ_64K/* Storage of test info to share with guest code */ struct test_config {
@@ -281,6 +286,101 @@ static void test_pre_fault_memory(unsigned long vm_type, enum vm_mem_backing_srcfor_each_guest_mode (__test_pre_fault_memory, &p); } +/* +* Pre-fault more dirty-logged pages than the ring holds: each call has to stop +* before the ring overflows, and no entry may be lost. +*/ +static void test_pre_fault_dirty_ring (void ) +{ +const u64 entries =TEST_RING_SIZE /sizeof (struct kvm_dirty_gfn ); +const u64 npages = entries + entries /4 ; +struct kvm_pre_fault_memory range, left; +struct kvm_dirty_gfn *gfns, *cur; +u64 harvested =0 , size; +struct kvm_vcpu *vcpu; +unsigned long *seen; +int ret, stops =0 ; +struct kvm_vm *vm; +u32 fetch =0 ; +gpa_t gpa; + +if (!kvm_has_cap (KVM_CAP_DIRTY_LOG_RING ) && !kvm_has_cap (KVM_CAP_DIRTY_LOG_RING_ACQ_REL )) { +pr_info ("Skipping the dirty ring test: no dirty ring\n" ); +return ; + } + + vm =vm_create (1 ); +vm_enable_dirty_ring (vm,TEST_RING_SIZE ); + vcpu =vm_vcpu_add (vm,0 ,NULL ); + gfns =vcpu_map_dirty_ring (vcpu); + +/* A ring entry is a host page. */ + size = npages *getpagesize (); + gpa =align_down (vm->max_gfn * vm->page_size - size,SZ_2M ); +vm_userspace_mem_region_add (vm,VM_MEM_SRC_ANONYMOUS , gpa,TEST_SLOT , +vm_num_guest_pages (vm->mode, npages),KVM_MEM_LOG_DIRTY_PAGES ); + +/* Populate the pages, so that pre-faulting can map them writable. */ +memset (addr_gpa2hva (vm, gpa),1 , size); + + seen =bitmap_zalloc (npages); +TEST_ASSERT (seen,"Failed to allocate the bitmap of reported pages" ); + + range = (struct kvm_pre_fault_memory ) { .gpa = gpa, .size = size }; +while (range.size) { +u64 n; + +vcpu_ioctl (vcpu,KVM_PRE_FAULT_MEMORY , &range); + +if (range.size) { + left = range; + ret =__vcpu_ioctl (vcpu,KVM_PRE_FAULT_MEMORY , &range); +TEST_ASSERT_VM_VCPU_IOCTL (ret ==-1 && errno ==ENOSPC , +KVM_PRE_FAULT_MEMORY , ret, vm); +TEST_ASSERT (range.gpa == left.gpa && range.size == left.size, +"KVM_PRE_FAULT_MEMORY made progress on a full ring" ); + stops++; + } + +for (n =0 ; ; n++) { + cur = &gfns[fetch % entries]; +/* Pairs with the smp_wmb() in kvm_dirty_ring_push(). */ +if (smp_load_acquire (&cur->flags) !=KVM_DIRTY_GFN_F_DIRTY ) +break ; + +TEST_ASSERT (cur->slot ==TEST_SLOT && cur->offset < npages, +"Ring entry %u: slot %u offset 0x%llx, expected slot %u offset < 0x%lx" , + fetch, cur->slot, cur->offset,TEST_SLOT , npages); +TEST_ASSERT (!__test_and_set_bit (cur->offset, seen), +"Ring entry %u: offset 0x%llx reported twice" , + fetch, cur->offset); +/* Pairs with kvm_dirty_gfn_harvested()'s smp_load_acquire(). */ +smp_store_release (&cur->flags,KVM_DIRTY_GFN_F_RESET ); + fetch++; + } + +/* A read pre-fault can map a page read-only, or access-tracked on x86. */ +if (!n && !harvested && !range.size) { +pr_info ("Skipping the dirty ring test: pre-faulting dirtied no pages\n" ); +bitmap_free (seen); +kvm_vm_free (vm); +return ; + } +TEST_ASSERT (n,"KVM_PRE_FAULT_MEMORY pushed nothing to the ring" ); +TEST_ASSERT (kvm_vm_reset_dirty_ring (vm) == n, +"KVM_RESET_DIRTY_RINGS didn't reset %lu entries" , n); + harvested += n; + } + +TEST_ASSERT (harvested == npages,"%lu ring entries for %lu pages" , harvested, npages); +TEST_ASSERT (stops,"KVM_PRE_FAULT_MEMORY never stopped for a %lu-entry ring" , entries); +pr_info ("Pre-faulted %lu pages through a %lu-entry ring, stopping %d times\n" , + npages, entries, stops); + +bitmap_free (seen); +kvm_vm_free (vm); +} +static void help (char *name) {puts ("" );
@@ -320,5 +420,6 @@ int main(int argc, char *argv[])test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , backing, false);test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , backing, true);#endif +test_pre_fault_dirty_ring ();return 0 ; }
Signature · 1 line
2.47.3
Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
In reply to Fuad TabbaQuoted 23 lines · Fuad Tabba
Pre-faulting can map a dirty-logged page writable, which pushes it to the vCPU's dirty ring.
KVM_RUNexits withKVM_EXIT_DIRTY_RING_FULLbefore the ring overflows, but the pre-fault loop never checks it, so one call can push more entries than the ring has free, which trips the WARN inkvm_dirty_ring_push()and overwrites entries userspace hasn't harvested. On x86:WARNING: arch/x86/kvm/../../../virt/kvm/dirty_ring.c:225 at kvm_dirty_ring_push+0x7a/0xd0 Call Trace: make_spte+0x2cc/0x450 kvm_tdp_mmu_map+0x5e1/0x7f0 kvm_tdp_page_fault+0x12c/0x1b0 kvm_mmu_do_page_fault+0x181/0x250 kvm_arch_vcpu_pre_fault_memory+0x136/0x1d0 kvm_vcpu_pre_fault_memory+0xcf/0x180 kvm_vcpu_ioctl+0x455/0x770Stop the loop once the ring is soft full, with the check
KVM_RUNmakes before entering the guest. Return the pages already processed, or-ENOSPCwhen there aren't any, since a retry can't succeed until userspace harvests and resets the ring. s390 has the ioctl but no ring, so givekvm_dirty_ring_check_request()a stub.
Quoted 6 lines
The check also sets
run->exit_reason, and arm64'sKVM_RUNcompletes a pending MMIO exit only ifexit_reasonis stillKVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the nextKVM_RUNdoesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit withKVM_RUNbefore it pre-faults.
That's also true for riscv:
switch (run->exit_reason) {
case KVM_EXIT_MMIO:
/* Process MMIO value returned from user-space */
ret = kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run);
break;
case KVM_EXIT_RISCV_SBI:
/* Process SBI value returned from user-space */
ret = kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run);
break;
case KVM_EXIT_RISCV_CSR:
/* Process CSR value returned from user-space */
ret = kvm_riscv_vcpu_csr_return(vcpu, vcpu->run);
break;
}(note that it skips CSR, SBI also with a changed exit_reason).
But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?
Quoted 34 lines
Fixes bc1a5cd002116 ("KVM: Add KVM_PRE_FAULT_MEMORY vcpu ioctl to pre-populate guest memory")Signed-off-by Fuad Tabba <[email protected]>--- Documentation/virt/kvm/api.rst | 9 +++++++-- include/linux/kvm_dirty_ring.h | 5 +++++ virt/kvm/kvm_main.c | 5 +++++ 3 files changed, 17 insertions(+), 2 deletions(-)Documentation/virt/kvm/api.rst +4
-1@@ -6544,6 +6544,7 @@ Errors: ENOENT The specified `gpa` is outside defined memslots. ENOEXEC The vCPU has not been initialised (arm64). EINTR An unmasked signal is pending and no page was processed. + ENOSPC The vCPU's dirty ring is full and no page was processed. EFAULT The parameter address was invalid. EHWPOISON A poisoned host page was encountered. EOPNOTSUPP Mapping memory for a GPA is unsupported by the@@ -6588,7 +6589,9 @@ case, the ioctl can be called in parallel. When the ioctl returns, the input values are updated to point to the remaining range. If `size` > 0 on return, the caller can just issue -the ioctl again with the same `struct kvm_map_memory` argument. +the ioctl again with the same `struct kvm_pre_fault_memory` argument. +If that fails with ENOSPC, the vCPU's dirty ring has to be harvested +and reset first. On x86, shadow page tables cannot support this ioctl because they are indexed by virtual address or nested guest physical address.
Documentation/virt/kvm/api.rst Therefore, the ioctl must be called *before* reading the content of · line 8965
Quoted 7 lines
Documentation/virt/kvm/api.rst +3
-1@@ -8957,7 +8960,9 @@ Therefore, the ioctl must be called *before* reading the content of the dirty pages. The dirty ring can get full. When it happens, the KVM_RUN of the -vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL. +vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and +KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the +ring has been harvested and reset.
Maybe worth mentioning the exit_reason thing if we keep this arch-generic?
Quoted 36 lines
Documentation/virt/kvm/api.rst +0
-0The dirty ring interface has a major difference comparing to the KVM_GET_DIRTY_LOG interface in that, when reading the dirty ring frominclude/linux/kvm_dirty_ring.h +5
-0@@ -60,6 +60,11 @@ static inline void kvm_dirty_ring_push(struct kvm_vcpu *vcpu, { } +static inline bool kvm_dirty_ring_check_request (struct kvm_vcpu *vcpu) +{ +return false; +} +static inline struct page *kvm_dirty_ring_get_page (struct kvm_dirty_ring *ring,u32 offset) {virt/kvm/kvm_main.c +5
-1@@ -4461,6 +4461,11 @@ static int kvm_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,break ; } +if (kvm_dirty_ring_check_request (vcpu)) { + r = -ENOSPC ; +break ; + } + r =kvm_arch_vcpu_pre_fault_memory (vcpu, range);if (WARN_ON_ONCE (r ==0 || r == -EIO ))break ; --
Quoted 2 lines
2.47.3
I wonder if this shouldn't just be an x86-only fix, since no other arch is impacted AFAICT.
arm64 and riscv never map a dirty-logged memslot writable, so they're never pushed to the rin, and s390 doesn't have a dirty ring at all:
arm64:
static int kvm_s2_fault_compute_prot(const struct kvm_s2_fault_desc *s2fd,
const struct kvm_s2_fault_vma_info *s2vi, enum kvm_pgtable_prot *prot)
{
...
if (s2vi->map_writable && (s2vi->device ||
!memslot_is_logging(s2fd->memslot) ||
kvm_s2_fault_is_write(s2fd)))
*prot |= KVM_PGTABLE_PROT_W;
...
}riscv:
int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot,
gpa_t gpa, unsigned long hva, bool is_write, struct kvm_gstage_mapping *out_map)
{
...
bool logging = kvm_slot_dirty_track_enabled(memslot) &&
!(memslot->flags & KVM_MEM_READONLY);
...
/*
* If logging is active then we allow writable pages only
* for write faults.
*/
if (logging && !is_write)
writable = false;
...
}An x86-only version would be pretty simple, something like the below.
This also covers off the retry (RET_PF_RETRY) loop which a generic version doesn't.
arch/x86/kvm/mmu/mmu.c +3 -0
@@ -5129,6 +5129,14 @@ static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa,if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu))return -EIO ; +if (kvm_dirty_ring_check_request (vcpu)) +return -ENOSPC ; + r =kvm_mmu_reload (vcpu);if (r)return r;
--
Cheers, Lorenzo
Re: [PATCH v2 2/2] KVM: selftests: Test KVM_PRE_FAULT_MEMORY against a full dirty ring
In reply to Fuad TabbaQuoted 8 lines · Fuad Tabba
Pre-fault more populated, dirty-logged pages than the vCPU's dirty ring holds, and check that each call stops before the ring overflows, that a retry returns ENOSPC without progress, and that each page is reported exactly once. A read pre-fault can map a page read-only, or access-tracked on x86 without EPT A/D bits, so skip the case if pre-faulting doesn't dirty any page.
Signed-off-by Fuad Tabba <[email protected]>
LGTM!
Quoted 137 lines
--- .../selftests/kvm/pre_fault_memory_test.c | 101 ++++++++++++++++++ 1 file changed, 101 insertions(+)tools/testing/selftests/kvm/pre_fault_memory_test.c +101
-1@@ -5,6 +5,9 @@* Author: * Isaku Yamahata <isaku.yamahata at gmail.com> */ +#include <asm/barrier.h> +#include <linux/bitmap.h> +#include <linux/bitops.h> #include <linux/sizes.h> #include <test_util.h> @@ -17,6 +20,8 @@/* Arbitrarily chosen values */ #define TEST_BASE_SIZE SZ_2M#define TEST_SLOT 10 +/* Large enough for the entries every architecture reserves. */ +#define TEST_RING_SIZE SZ_64K/* Storage of test info to share with guest code */ struct test_config {@@ -281,6 +286,101 @@ static void test_pre_fault_memory(unsigned long vm_type, enum vm_mem_backing_srcfor_each_guest_mode (__test_pre_fault_memory, &p); } +/* +* Pre-fault more dirty-logged pages than the ring holds: each call has to stop +* before the ring overflows, and no entry may be lost. +*/ +static void test_pre_fault_dirty_ring (void ) +{ +const u64 entries =TEST_RING_SIZE /sizeof (struct kvm_dirty_gfn ); +const u64 npages = entries + entries /4 ; +struct kvm_pre_fault_memory range, left; +struct kvm_dirty_gfn *gfns, *cur; +u64 harvested =0 , size; +struct kvm_vcpu *vcpu; +unsigned long *seen; +int ret, stops =0 ; +struct kvm_vm *vm; +u32 fetch =0 ; +gpa_t gpa; + +if (!kvm_has_cap (KVM_CAP_DIRTY_LOG_RING ) && !kvm_has_cap (KVM_CAP_DIRTY_LOG_RING_ACQ_REL )) { +pr_info ("Skipping the dirty ring test: no dirty ring\n" ); +return ; + } + + vm =vm_create (1 ); +vm_enable_dirty_ring (vm,TEST_RING_SIZE ); + vcpu =vm_vcpu_add (vm,0 ,NULL ); + gfns =vcpu_map_dirty_ring (vcpu); + +/* A ring entry is a host page. */ + size = npages *getpagesize (); + gpa =align_down (vm->max_gfn * vm->page_size - size,SZ_2M ); +vm_userspace_mem_region_add (vm,VM_MEM_SRC_ANONYMOUS , gpa,TEST_SLOT , +vm_num_guest_pages (vm->mode, npages),KVM_MEM_LOG_DIRTY_PAGES ); + +/* Populate the pages, so that pre-faulting can map them writable. */ +memset (addr_gpa2hva (vm, gpa),1 , size); + + seen =bitmap_zalloc (npages); +TEST_ASSERT (seen,"Failed to allocate the bitmap of reported pages" ); + + range = (struct kvm_pre_fault_memory ) { .gpa = gpa, .size = size }; +while (range.size) { +u64 n; + +vcpu_ioctl (vcpu,KVM_PRE_FAULT_MEMORY , &range); + +if (range.size) { + left = range; + ret =__vcpu_ioctl (vcpu,KVM_PRE_FAULT_MEMORY , &range); +TEST_ASSERT_VM_VCPU_IOCTL (ret ==-1 && errno ==ENOSPC , +KVM_PRE_FAULT_MEMORY , ret, vm); +TEST_ASSERT (range.gpa == left.gpa && range.size == left.size, +"KVM_PRE_FAULT_MEMORY made progress on a full ring" ); + stops++; + } + +for (n =0 ; ; n++) { + cur = &gfns[fetch % entries]; +/* Pairs with the smp_wmb() in kvm_dirty_ring_push(). */ +if (smp_load_acquire (&cur->flags) !=KVM_DIRTY_GFN_F_DIRTY ) +break ; + +TEST_ASSERT (cur->slot ==TEST_SLOT && cur->offset < npages, +"Ring entry %u: slot %u offset 0x%llx, expected slot %u offset < 0x%lx" , + fetch, cur->slot, cur->offset,TEST_SLOT , npages); +TEST_ASSERT (!__test_and_set_bit (cur->offset, seen), +"Ring entry %u: offset 0x%llx reported twice" , + fetch, cur->offset); +/* Pairs with kvm_dirty_gfn_harvested()'s smp_load_acquire(). */ +smp_store_release (&cur->flags,KVM_DIRTY_GFN_F_RESET ); + fetch++; + } + +/* A read pre-fault can map a page read-only, or access-tracked on x86. */ +if (!n && !harvested && !range.size) { +pr_info ("Skipping the dirty ring test: pre-faulting dirtied no pages\n" ); +bitmap_free (seen); +kvm_vm_free (vm); +return ; + } +TEST_ASSERT (n,"KVM_PRE_FAULT_MEMORY pushed nothing to the ring" ); +TEST_ASSERT (kvm_vm_reset_dirty_ring (vm) == n, +"KVM_RESET_DIRTY_RINGS didn't reset %lu entries" , n); + harvested += n; + } + +TEST_ASSERT (harvested == npages,"%lu ring entries for %lu pages" , harvested, npages); +TEST_ASSERT (stops,"KVM_PRE_FAULT_MEMORY never stopped for a %lu-entry ring" , entries); +pr_info ("Pre-faulted %lu pages through a %lu-entry ring, stopping %d times\n" , + npages, entries, stops); + +bitmap_free (seen); +kvm_vm_free (vm); +} +static void help (char *name) {puts ("" );@@ -320,5 +420,6 @@ int main(int argc, char *argv[])test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , backing, false);test_pre_fault_memory (KVM_X86_SW_PROTECTED_VM , backing, true);#endif +test_pre_fault_dirty_ring ();return 0 ; } --
Quoted 2 lines
2.47.3
--
Cheers, Lorenzo
Fuad Tabba
reviewerRe: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
In reply to Lorenzo Stoakes (ARM)Hi Lorenzo,
On Tue, 17 Nov 2026 14:23:09 +0000, "Lorenzo Stoakes (ARM)" <[email protected]> wrote:
[...]
Quoted 24 lines
Quoted 6 lines
The check also sets
run->exit_reason, and arm64'sKVM_RUNcompletes a pending MMIO exit only ifexit_reasonis stillKVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the nextKVM_RUNdoesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit withKVM_RUNbefore it pre-faults.That's also true for riscv:
switch (run->exit_reason) { case KVM_EXIT_MMIO: /* Process MMIO value returned from user-space */ ret = kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run); break; case KVM_EXIT_RISCV_SBI: /* Process SBI value returned from user-space */ ret = kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run); break; case KVM_EXIT_RISCV_CSR: /* Process CSR value returned from user-space */ ret = kvm_riscv_vcpu_csr_return(vcpu, vcpu->run); break; }
Quoted 1 line
(note that it skips CSR, SBI also with a changed
exit_reason).
Ack.
Quoted 2 lines
But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?
The ring overflows on arm64 too (below). As for exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between a KVM_EXIT_MMIO return and the next KVM_RUN "gets to deal with the fallout" [1]. I think the same goes for riscv's exits.
[...]
Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0
-0
[...]
Quoted 7 lines
Documentation/virt/kvm/api.rst
Quoted 5 lines
The dirty ring can get full. When it happens, the
KVM_RUNof theDocumentation/virt/kvm/api.rst +3
-1-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL. +vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and +KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the +ring has been harvested and reset.Maybe worth mentioning the
exit_reasonthing if we keep this arch-generic?
Sure, I'll document that userspace should complete a pending exit with KVM_RUN before it pre-faults on that vCPU.
[...]
Quoted 11 lines
I wonder if this shouldn't just be an x86-only fix, since no other arch is impacted AFAICT.
arm64 and riscv never map a dirty-logged memslot writable, so they're never pushed to the rin, and s390 doesn't have a dirty ring at all:
arm64:
static int kvm_s2_fault_compute_prot(const struct
kvm_s2_fault_desc*s2fd,
const structkvm_s2_fault_vma_info*s2vi, enumkvm_pgtable_prot*prot)
Quoted 8 lines
{
...
if (s2vi->map_writable && (s2vi->device ||
!memslot_is_logging(s2fd->memslot) ||
kvm_s2_fault_is_write(s2fd)))
*prot |= KVM_PGTABLE_PROT_W;
...
}arm64 does map a logged slot as writable today. memslot_is_logging() tests memslot->dirty_bitmap, which isn't allocated when only the dirty ring is enabled, and patch 2's selftest triggers the WARN in kvm_dirty_ring_push() on arm64 without patch 1. Wei-Lin's fix to memslot_is_logging() [2] is in kvmarm/fixes, not in mainline yet.
With it, the s2vi->device case above still maps a device page writable on a read fault in a logged slot. kvm_arch_prepare_memory_region() rejects logging on a VM_PFNMAP VMA, so that requires userspace to map the device under the slot after creating it (I haven't tested it).
So I think the check should be in the generic loop.
[...]
Quoted 4 lines
An x86-only version would be pretty simple, something like the below.
This also covers off the retry (
RET_PF_RETRY) loop which a generic version doesn't.
True. To overflow there, the retries for one page would have to push more entries than the ring has in reserve (at least 64), and I don't see a way to get that, so I left it alone.
Cheers,
/fuad
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Sean Christopherson
maintainerRe: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
Quoted 20 lines · Fuad Tabba
Quoted 2 lines
But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?
The ring overflows on arm64 too (below). As for
exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between aKVM_EXIT_MMIOreturn and the nextKVM_RUN"gets to deal with the fallout" [1]. I think the same goes for riscv's exits.[...]
Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0
-0[...]
Quoted 7 lines
Documentation/virt/kvm/api.rst
Quoted 5 lines
The dirty ring can get full. When it happens, the
KVM_RUNof theDocumentation/virt/kvm/api.rst +3
-1-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL. +vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and +KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the +ring has been harvested and reset.Maybe worth mentioning the
exit_reasonthing if we keep this arch-generic?
Quoted 2 lines
Sure, I'll document that userspace should complete a pending exit with
KVM_RUNbefore it pre-faults on that vCPU.
If we're going to do a documentation update, I vote to keep it vague and scary, otherwise readers might think the exception proves the rule. I.e. if we say that KVM_PRE_FAULT_MEMORY isn't supported, then userspace might think all other ioctls are safe, which is most definitely not the case. E.g. something like
Documentation/virt/kvm/api.rst +7 -4
@@ -6971,10 +6971,13 @@ to the byte array. The pending state of the operation is not preserved in state which is visible to userspace, thus userspace should ensure that the operation is - completed before performing a live migration. Userspace can re-enter the - guest with an unmasked signal pending or with the immediate_exit field set - to complete pending operations without allowing any further instructions - to be executed. + completed before performing a live migration. In general, userspace should + exercise extreme caution when invoking ioctls other than KVM_RUN prior to + completing the original operation. + + Userspace can re-enter the guest with an unmasked signal pending or with + the immediate_exit field set to complete pending operations without + allowing any further instructions to be executed. ::
Fuad Tabba
reviewerRe: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
Hi Sean,
On Thu, 19 Nov 2026 at 00:25, Sean Christopherson <[email protected]> wrote:
[...]
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
Maybe worth mentioning the
exit_reasonthing if we keep this arch-generic?Sure, I'll document that userspace should complete a pending exit with
KVM_RUNbefore it pre-faults on that vCPU.If we're going to do a documentation update, I vote to keep it vague and scary, otherwise readers might think the exception proves the rule. I.e. if we say that
KVM_PRE_FAULT_MEMORYisn't supported, then userspace might think all other ioctls are safe, which is most definitely not the case. E.g. something like
I'll go with that in v3.
Thanks,
/fuad
Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
In reply to Fuad TabbaQuoted 29 lines · Fuad Tabba
Hi Lorenzo,
On Tue, 17 Nov 2026 14:23:09 +0000, "Lorenzo Stoakes (ARM)" <[email protected]> wrote:
[...]Quoted 24 lines
Quoted 6 lines
The check also sets
run->exit_reason, and arm64'sKVM_RUNcompletes a pending MMIO exit only ifexit_reasonis stillKVM_EXIT_MMIO. If a pre-fault stops on a full ring while an MMIO exit is pending, the nextKVM_RUNdoesn't complete the exit and the guest repeats the MMIO access. Userspace can avoid that by completing the exit withKVM_RUNbefore it pre-faults.That's also true for riscv:
switch (run->exit_reason) { case KVM_EXIT_MMIO: /* Process MMIO value returned from user-space */ ret = kvm_riscv_vcpu_mmio_return(vcpu, vcpu->run); break; case KVM_EXIT_RISCV_SBI: /* Process SBI value returned from user-space */ ret = kvm_riscv_vcpu_sbi_return(vcpu, vcpu->run); break; case KVM_EXIT_RISCV_CSR: /* Process CSR value returned from user-space */ ret = kvm_riscv_vcpu_csr_return(vcpu, vcpu->run); break; }
Quoted 3 lines
Quoted 1 line
(note that it skips CSR, SBI also with a changed
exit_reason).Ack.
Probably worth referencing in commit msg I guess?
Quoted 9 lines
Quoted 3 lines
But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?
The ring overflows on arm64 too (below). As for
exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between aKVM_EXIT_MMIOreturn and the nextKVM_RUN"gets to deal with the fallout" [1]. I think the same goes for riscv's exits.
I mean generally yeah, users doing silly things can expect silly results I guess! :)
Quoted 12 lines
[...]
Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0
-0[...]
Quoted 7 lines
Documentation/virt/kvm/api.rst
Quoted 5 lines
The dirty ring can get full. When it happens, the
KVM_RUNof theDocumentation/virt/kvm/api.rst +3
-1-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL. +vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and +KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the +ring has been harvested and reset.Maybe worth mentioning the
exit_reasonthing if we keep this arch-generic?
Quoted 2 lines
Sure, I'll document that userspace should complete a pending exit with
KVM_RUNbefore it pre-faults on that vCPU.
Thanks!
Quoted 22 lines
[...]
Quoted 11 lines
I wonder if this shouldn't just be an x86-only fix, since no other arch is impacted AFAICT.
arm64 and riscv never map a dirty-logged memslot writable, so they're never pushed to the rin, and s390 doesn't have a dirty ring at all:
arm64:
static int kvm_s2_fault_compute_prot(const struct
kvm_s2_fault_desc*s2fd,
const structkvm_s2_fault_vma_info*s2vi, enumkvm_pgtable_prot*prot)Quoted 8 lines
{ ... if (s2vi->map_writable && (s2vi->device || !memslot_is_logging(s2fd->memslot) || kvm_s2_fault_is_write(s2fd))) *prot |= KVM_PGTABLE_PROT_W; ... }
Quoted 5 lines
arm64 does map a logged slot as writable today.
memslot_is_logging()testsmemslot->dirty_bitmap, which isn't allocated when only the dirty ring is enabled, and patch 2's selftest triggers the WARN inkvm_dirty_ring_push()on arm64 without patch 1. Wei-Lin's fix tomemslot_is_logging()[2] is in kvmarm/fixes, not in mainline yet.
But Wei-Lin's fix was taken? So I don't think predicating things on the basis of it not being there really makes any sense.
However...
Quoted 5 lines
With it, the
s2vi->devicecase above still maps a device page writable on a read fault in a logged slot.kvm_arch_prepare_memory_region()rejects logging on aVM_PFNMAPVMA, so that requires userspace to map the device under the slot after creating it (I haven't tested it).
...Given that this possibility exists and device pages are unconditionally marked writable (even if it's mildly contrived to get there), it's fair enough to put it in the generic code.
Quoted 4 lines
So I think the check should be in the generic loop.
[...]
Quoted 8 lines
Quoted 4 lines
An x86-only version would be pretty simple, something like the below.
This also covers off the retry (
RET_PF_RETRY) loop which a generic version doesn't.True. To overflow there, the retries for one page would have to push more entries than the ring has in reserve (at least 64), and I don't see a way to get that, so I left it alone.
Yep that's fair enough!
Quoted 6 lines
Cheers,
/fuad[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
--
Cheers, Lorenzo
Re: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
In reply to Sean ChristophersonQuoted 21 lines · Sean Christopherson
Quoted 20 lines · Fuad Tabba
Quoted 2 lines
But the overflow bug only applies to x86 (see below), so is this something we should worry about at all?
The ring overflows on arm64 too (below). As for
exit_reason, I asked Sean about the arm64 MMIO case on v1, and his answer was that a VMM that pre-faults between aKVM_EXIT_MMIOreturn and the nextKVM_RUN"gets to deal with the fallout" [1]. I think the same goes for riscv's exits.[...]
Quoted 1 line
Quoted 1 line
Documentation/virt/kvm/api.rst +0
-0[...]
Quoted 7 lines
Documentation/virt/kvm/api.rst
Quoted 5 lines
The dirty ring can get full. When it happens, the
KVM_RUNof theDocumentation/virt/kvm/api.rst +3
-1-vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL. +vcpu will return with exit reason KVM_EXIT_DIRTY_RING_FULL, and +KVM_PRE_FAULT_MEMORY on the vcpu will process no more pages until the +ring has been harvested and reset.Maybe worth mentioning the
exit_reasonthing if we keep this arch-generic?
Quoted 7 lines
Quoted 2 lines
Sure, I'll document that userspace should complete a pending exit with
KVM_RUNbefore it pre-faults on that vCPU.If we're going to do a documentation update, I vote to keep it vague and scary, otherwise readers might think the exception proves the rule. I.e. if we say that
KVM_PRE_FAULT_MEMORYisn't supported, then userspace might think all other ioctls are safe, which is most definitely not the case. E.g. something like
Yes definitely, like drivers, users can get up to all sorts of mischief if not properly forewarned :>)
Quoted 12 lines
Documentation/virt/kvm/api.rst +0
-3@@ -6971,10 +6971,13 @@ to the byte array. The pending state of the operation is not preserved in state which is visible to userspace, thus userspace should ensure that the operation is - completed before performing a live migration. Userspace can re-enter the - guest with an unmasked signal pending or with the immediate_exit field set - to complete pending operations without allowing any further instructions
Documentation/virt/kvm/api.rst to the byte array. · line 6982
Quoted 10 lines
Documentation/virt/kvm/api.rst +7
-1- to be executed. + completed before performing a live migration. In general, userspace should + exercise extreme caution when invoking ioctls other than KVM_RUN prior to + completing the original operation. + + Userspace can re-enter the guest with an unmasked signal pending or with + the immediate_exit field set to complete pending operations without + allowing any further instructions to be executed. ::
--
Cheers, Lorenzo
Fuad Tabba
reviewerRe: [PATCH v2 1/2] KVM: Stop KVM_PRE_FAULT_MEMORY when the vCPU's dirty ring is full
Hi Lorenzo,
On Thu, 19 Nov 2026 11:18:48 +0000, "Lorenzo Stoakes (ARM)" <[email protected]> wrote:
[...]
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
(note that it skips CSR, SBI also with a changed
exit_reason).Ack.
Probably worth referencing in commit msg I guess?
Will add one in v3.
Cheers,
/fuad