kvm: add EVER_MAPPED ioctl and capability
Subsystems (5)
PATCH 1/1 kvm: add EVER_MAPPED ioctl and capability
This allows a VMM to keep track of which pages were mapped by the guest at any point. The main use case for this is supporting the HvExtCallGetBootZeroedMemory Hyper-V hypercall, which allows a guest to inquire which of its pages are already zeroed (and so don't need zeroing again inside the guest).
This is implemented by setting up a bitmap that tracks memory at a 2MB granularity. Whenever a leaf page is created in the spt, the corresponding bit is set to 1; values are never reset. So even if entries are zapped, we still knows which pages were ever mapped, a conservative estimate of which pages were ever modified. The bitmap has to be kept for the lifetime of the VM, but the 2MB granularity means the size is small: even a 16TB VM would only have a bitmap size of 1MB.
Note that for a VMM to properly support the hypercall, it also has to track all the pages it itself touches on behalf of the VM; this is clearly beyond the scope of this patch, but it is on the VMM to merge these two sets of information in some way.
--- Documentation/virt/kvm/api.rst | 67 ++++++ arch/x86/include/asm/kvm_host.h | 9 + arch/x86/kvm/mmu/ever_mapped.h | 33 +++ arch/x86/kvm/mmu/tdp_mmu.c | 12 ++ arch/x86/kvm/x86.c | 110 ++++++++++ include/uapi/linux/kvm.h | 15 ++ tools/testing/selftests/kvm/Makefile.kvm | 1 + .../testing/selftests/kvm/ever_mapped_test.c | 191 ++++++++++++++++++ 8 files changed, 438 insertions(+) create mode 100644 arch/x86/kvm/mmu/ever_mapped.h create mode 100644 tools/testing/selftests/kvm/ever_mapped_test.c
Documentation/virt/kvm/api.rst +67 -0
@@ -6566,6 +6566,53 @@ KVM_S390_KEYOP_SSKE Sets the storage key for the guest address ``guest_addr`` to the key specified in ``key``, returning the previous value in ``key``. +4.145 KVM_GET_EVER_MAPPED_LOG +----------------------------- + +:Capability: KVM_CAP_EVER_MAPPED +:Architectures: x86 +:Type: vm ioctl +:Parameters: struct kvm_ever_mapped_log +:Returns: 0 on success, -ERRNO on error + +Errors: + + ====== ============================================================ + ENOENT KVM_CAP_EVER_MAPPED has not been enabled for this VM + EINVAL ``granule_shift`` is below 21 (internal tracking granularity) or + above 63 + EINVAL ``first_granule`` + ``num_granules`` extendends beyond the maxgpa + set up when KVM_CAP_EVER_MAPPED was enabled + EINVAL ``flags`` is not 0 + EFAULT copy_{from,to}_user of ``bitmap`` failed, e.g., bitmap wasn't + sufficiently sized + ENOMEM setting up temporary bitmap for copying to userspace failed + ====== ============================================================ + +This requires KVM_CAP_EVER_MAPPED to have been enabled for the VM with a maxgpa +parameter. + +:: + + struct kvm_ever_mapped_log { + __u64 first_granule; + __u64 num_granules; + __u32 granule_shift; + __u32 flags; + union { + void __user *bitmap; + __u64 padding; + }; + __u64 reserved[4]; + }; + +``first_granule``, ``num_granules``, and ``granule_shift`` encode the GPA range +the caller inquires about. For example, values of 2, 25, and 21, respectively, +inquire about the address range [4MB, 54MB). The caller provides a ``bitmap`` +of the required length (in the above case 4 bytes). The call will write a 1 if +any memory in the granule has been mapped at some point since +KVM_CAP_EVER_MAPPED was enabled, 0 if not. + .. _kvm_run: 5. The kvm_run structure
@@ -8949,6 +8996,26 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key interpretation are disabled. If cmma has already been enabled or the hpage_2g module parameter is not set to 1, -EINVAL is returned. +7.48 KVM_CAP_EVER_MAPPED +------------------------ + +:Architectures: x86 +:Parameters: args[0] - maximum GPA to track mapped memory +:Returns: 0 on success, -EOPNOTSUPP if TDP is not available, -EINVAL if args[0] + is 0 or beyond the maximum possible GPA, -ENOMEM if tracking setup + failed, -EEXIST if already set up. + +The presence of this capability indicates that KVM can track if pages have ever +been mapped by the guest. This allows a conservative estimate of which pages +are not zero any more. + +On enabling the capability, the caller provides a max GPA value. KVM will then +track GPAs between 0 and this value. Note that this capability can be enabled +at any time, but it is strongly recommended to enable it before vCPUs start +running. Any page accesses by the guest before capability enablement will not +be tracked. The max GPA value can only be set once; subsequent attempts to +enable the capability with a different or even same value will fail. + 8. Other capabilities. ======================
arch/x86/include/asm/kvm_host.h +9 -0
@@ -1533,6 +1533,15 @@ struct kvm_arch {*/ bool shadow_root_allocated; +/* +* Tracks which guest pages have ever been mapped. This allows +* tracking which memory is guaranteed to still be zero, e.g., for +* supporting the HvExtCallGetBootZeroedMemory enlightenment. +* The granularity is defined by KVM_EVER_MAPPED_SHIFT. +*/ +unsigned long *ever_mapped_bitmap; +u64 ever_mapped_max_gpa; +#ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING /* * If set, the VM has (or had) an external write tracking user, and
arch/x86/kvm/mmu/ever_mapped.h new file +33 -0
@@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_EVER_MAPPED_H +#define __KVM_X86_EVER_MAPPED_H + +#include <linux/kvm_host.h> + +/* +* Track if pages have ever been mapped at an internal granularity of 2 MB. +*/ +#define KVM_EVER_MAPPED_SHIFT 21 + +static inline void kvm_ever_mapped_set_range (struct kvm *kvm,gfn_t gfn, +u64 npages) +{ +const unsigned int shift =KVM_EVER_MAPPED_SHIFT -PAGE_SHIFT ; +unsigned long start = gfn >> shift; +unsigned long end = (gfn + npages + (1UL << shift) -1 ) >> shift; +unsigned long i; + + end =min (end, kvm->arch.ever_mapped_max_gpa >>KVM_EVER_MAPPED_SHIFT ); +for (i = start; i < end; i++) { +/* +* re-mapping zapped pages and breaking up of huge pages can +* trigger a number of sets on already set bits, so test first +* before atomic-set. Since bits can never be unset again, this +* is safe against races. +*/ +if (!test_bit (i, kvm->arch.ever_mapped_bitmap)) +set_bit (i, kvm->arch.ever_mapped_bitmap); + } +} + +#endif /* __KVM_X86_EVER_MAPPED_H */
arch/x86/kvm/mmu/tdp_mmu.c +12 -0
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 #define pr_fmt (fmt) KBUILD_MODNAME ": " fmt +#include "ever_mapped.h" #include "mmu.h" #include "mmu_internal.h" #include "mmutrace.h"
@@ -554,6 +555,17 @@ static int __handle_changed_spte(struct kvm *kvm, struct kvm_mmu_page *sp,return 0 ; } +/* +* Whenever a page becomes a fresh leaf, mark the memory as mapped. +* On hugepage breakup, this triggers for each new lower-level page, +* even though the memory was already marked when the hugepage was +* mapped, but the overhead is negligible. +*/ +if (unlikely (kvm->arch.ever_mapped_bitmap) && + is_leaf && !was_leaf) { +kvm_ever_mapped_set_range (kvm, gfn,KVM_PAGES_PER_HPAGE (level)); + } +/* * Recursively handle child PTs if the change removed a subtree from * the paging structure. Note the WARN on the PFN changing without the
arch/x86/kvm/x86.c +110 -0
140 lines not on this page · raw message
include/uapi/linux/kvm.h +15 -0
@@ -997,6 +997,7 @@ struct kvm_enable_cap {#define KVM_CAP_S390_KEYOP 247#define KVM_CAP_S390_VSIE_ESAMODE 248#define KVM_CAP_S390_HPAGE_2G 249 +#define KVM_CAP_EVER_MAPPED 250struct kvm_irq_routing_irqchip {__u32 irqchip;
@@ -1670,4 +1671,18 @@ struct kvm_pre_fault_memory { __u64 padding[5]; }; +#define KVM_GET_EVER_MAPPED_LOG _IOW(KVMIO, 0xd6, struct kvm_ever_mapped_log) + +struct kvm_ever_mapped_log { +__u64 first_granule; +__u64 num_granules; +__u32 granule_shift; +__u32 flags; +union { +void __user *bitmap; +__u64 padding; + }; +__u64 reserved[4 ]; +}; +#endif /* __LINUX_KVM_H */
tools/testing/selftests/kvm/Makefile.kvm +1 -0
@@ -162,6 +162,7 @@ TEST_GEN_PROGS_x86 += rseq_test TEST_GEN_PROGS_x86 += steal_time TEST_GEN_PROGS_x86 += system_counter_offset_test TEST_GEN_PROGS_x86 += pre_fault_memory_test +TEST_GEN_PROGS_x86 += ever_mapped_test # Compiled outputs used by test targets TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
tools/testing/selftests/kvm/ever_mapped_test.c new file +191 -0
191 lines not on this page · raw message
Signature · 1 line
2.54.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider: - [High] The `ever_mapped_bitmap` is dynamically allocated during VM capability enablement but never freed…
Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [High] The ever_mapped_bitmap is dynamically allocated during VM capability enablement but never freed during VM teardown.
- [High] Buffer overflow and KASAN out-of-bounds access due to a byte-aligned allocation size being manipulated by unsigned long bit operations.
- [High] Missing cond_resched() in the fallback iteration loop allows a single ioctl to monopolize the CPU, causing soft lockups.
- [High] KVM_CAP_EVER_MAPPED enablement succeeds based on the global tdp_enabled hardware flag, but silently fails to track memory if the legacy shadow MMU is actively handling the VM.
- [Medium] Unvalidated reserved fields in struct kvm_ever_mapped_log break future ABI compatibility.
- [Medium] Lockless reading of kvm->arch.ever_mapped_max_gpa and ever_mapped_bitmap can cause TOCTOU races and spurious errors.
--
--- Patch [1]: [PATCH] kvm: add EVER_MAPPED ioctl and capability --- commit bb0c368d3b83d1838e5cca827b1430b4b426ca4d
Author: Florian Schmidt <[email protected]>
Subject: kvm: add EVER_MAPPED ioctl and capability
This commit introduces a new capability, KVM_CAP_EVER_MAPPED, which allows a VMM to track which guest pages have ever been mapped using a bitmap. It also adds an ioctl, KVM_GET_EVER_MAPPED_LOG, to retrieve this tracking information.
Quoted 4 lines
arch/x86/kvm/x86.c +0
-0
[ ... ]
arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2397
Quoted 7 lines
arch/x86/kvm/x86.c +3
-0@@ -2391,6 +2392,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_READONLY_MEM : r = kvm ?kvm_arch_has_readonly_mem (kvm) :1 ;break ; +case KVM_CAP_EVER_MAPPED : + r = (tdp_enabled &&IS_ENABLED (CONFIG_X86_64 )); +break ;
[Severity: High]
Does this successfully track memory when the VM is running under the legacy shadow MMU?
If a VM uses the legacy shadow MMU (for example, with nested L2 guests or the kvm.tdp_mmu=0 module parameter), the page mappings are handled in mmu.c, which bypasses the kvm_ever_mapped_set_range() hook added to tdp_mmu.c. Since this check only verifies the global tdp_enabled hardware flag, could this silently fail to track memory and cause userspace to observe a clean log when the pages were actually modified?
[ ... ]
Quoted 12 lines
arch/x86/kvm/x86.c +8
-0@@ -4190,6 +4194,36 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,mutex_unlock (&kvm->lock);break ; } +case KVM_CAP_EVER_MAPPED : { +unsigned long *mapped_bitmap; +unsigned long num_granules; + + r =0 ; +mutex_lock (&kvm->lock); +if (!tdp_enabled) { + r = -EOPNOTSUPP ;
arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4214
Quoted 10 lines
arch/x86/kvm/x86.c +10
-0+ }else if (kvm->arch.ever_mapped_bitmap) { + r = -EEXIST ; + }else if (cap->args[0 ] ==0 || + cap->args[0 ] > (1ULL << kvm_host.maxphyaddr)) { + r = -EINVAL ; + }else { + num_granules =DIV_ROUND_UP (cap->args[0 ],1ULL <<KVM_EVER_MAPPED_SHIFT ); + mapped_bitmap = +kvcalloc (BITS_TO_LONGS (num_granules), +sizeof (long ),GFP_KERNEL_ACCOUNT );
[Severity: High]
Is this allocated memory ever freed?
When a VM is destroyed via the kvm_arch_destroy_vm() path, does it clean up the ever_mapped_bitmap? An unprivileged process could repeatedly create VMs, enable this capability, and destroy the VMs, potentially exhausting host memory if the bitmap isn't explicitly freed.
Quoted 12 lines
arch/x86/kvm/x86.c +12
-0+if (!mapped_bitmap) + r = -ENOMEM ; +else { +write_lock (&kvm->mmu_lock); + kvm->arch.ever_mapped_bitmap = mapped_bitmap; + kvm->arch.ever_mapped_max_gpa = num_granules <<KVM_EVER_MAPPED_SHIFT ; +write_unlock (&kvm->mmu_lock); + } + } +mutex_unlock (&kvm->lock); +break ; + }
[ ... ]
Quoted 7 lines
arch/x86/kvm/x86.c +3
-0@@ -4197,6 +4231,73 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,return r; } +static int kvm_vm_ioctl_get_ever_mapped_log (struct kvm *kvm, +struct kvm_ever_mapped_log *log) +{
arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4246
Quoted 10 lines
arch/x86/kvm/x86.c +10
-0+u64 max_granules; +unsigned int coarseness; +unsigned long *bitmap; +int r; + +if (!kvm->arch.ever_mapped_bitmap) +return -ENOENT ; + +if (log->flags) +return -EINVAL ;
[Severity: Medium]
Should we also validate that the reserved fields in the log structure are strictly zero here?
If userspace leaves log->reserved uninitialized, stack garbage could be passed in. Does this prevent safely repurposing these fields in the future without breaking ABI compatibility with older binaries?
arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4251
Quoted 5 lines
arch/x86/kvm/x86.c +5
-0+ +if (log->granule_shift <KVM_EVER_MAPPED_SHIFT || log->granule_shift >=64 ) +return -EINVAL ; + + max_granules = kvm->arch.ever_mapped_max_gpa >> log->granule_shift;
[Severity: Medium]
Could this lockless read of ever_mapped_max_gpa cause a TOCTOU race?
If this is called concurrently with KVM_CAP_EVER_MAPPED enablement, could the reader observe a non-NULL ever_mapped_bitmap but a stale max_gpa of 0 due to compiler reordering or missing memory barriers?
If so, max_granules would evaluate to 0 and the ioctl might erroneously fail with -EINVAL.
Quoted 14 lines
arch/x86/kvm/x86.c +14
-0+if (log->num_granules > max_granules || + log->first_granule > max_granules - log->num_granules) +return -EINVAL ; + +/* +* For the case where the request uses the same granularity as our internal +* tracking, and we are byte-aligned (the expected common case), we can +* just copy_to_user. Otherwise, we have to create a new bitmap to pass. +* +* We do not lock against concurrent vCPUs mapping new memory on any of +* these operations. Bits are set atomically and never cleared, so any race +* here is indistinguishable from a write happening after this handler +* finishes but before the caller reads the results. +*/
arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4275
Quoted 10 lines
arch/x86/kvm/x86.c +10
-0+ coarseness = log->granule_shift -KVM_EVER_MAPPED_SHIFT ; +if (coarseness ==0 && !(log->first_granule &7 ) && !(log->num_granules &7 )) { +if (copy_to_user (log->bitmap, + (u8 *)kvm->arch.ever_mapped_bitmap + log->first_granule /8 , + log->num_granules /8 )) +return -EFAULT ; +return 0 ; + } + + bitmap =kvzalloc (DIV_ROUND_UP (log->num_granules,8 ),GFP_KERNEL_ACCOUNT );
[Severity: High]
Does this allocation size safely accommodate the unsigned long bit operations that follow?
The buffer is allocated with exact byte granularity here using kvzalloc(DIV_ROUND_UP(log->num_granules, 8), ...), but it is later populated using __set_bit() which operates on 8-byte (unsigned long) boundaries.
If the requested size is not a multiple of 8 bytes (for example, 9 bytes for 65 granules), could __set_bit() access memory past the requested allocation boundary and trigger a KASAN out-of-bounds splat?
arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4281
Quoted 6 lines
arch/x86/kvm/x86.c +6
-0+if (!bitmap) +return -ENOMEM ; + +for (u64 gran = log->first_granule; + gran < log->first_granule + log->num_granules; + gran++) {
[Severity: High]
Can this loop monopolize the CPU and trigger a soft lockup?
If userspace calls this ioctl with unaligned parameters that bypass the copy_to_user fast path, log->num_granules can be very large (up to 2^31 for a 52-bit physical address space and a 2MB granule).
Does iterating over a potentially massive user-controlled scale without a cond_resched() in this loop risk keeping the CPU occupied long enough to trigger the soft lockup watchdog?
Quoted 6 lines
arch/x86/kvm/x86.c +6
-0+unsigned long start = gran << coarseness; +unsigned long end = start + (1UL << coarseness); + +if (find_next_bit (kvm->arch.ever_mapped_bitmap, end, start) < end) +__set_bit (gran - log->first_granule, bitmap); + }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Sean Christopherson
maintainerQuoted 2 lines · Florian Schmidt
This allows a VMM to keep track of which pages were mapped by the guest at any point.
So does userfaultfd. Any performance concerns with UFFD should be addressed by KVM Userfault, which is on my todo list for this year.