Skip to contentSkip to conversation

kvm: add EVER_MAPPED ioctl and capability

PATCH In review Florian Schmidt 3 participants 3 messages Jul 31

Subsystems (5)
Documentation x86 KVM Kselftest +1 more subsystems
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 kvm: add EVER_MAPPED ioctl and capability

This allows a VMM to keep track of which pages were mapped by the guest at any point. The main use case for this is supporting the HvExtCallGetBootZeroedMemory Hyper-V hypercall, which allows a guest to inquire which of its pages are already zeroed (and so don't need zeroing again inside the guest).

This is implemented by setting up a bitmap that tracks memory at a 2MB granularity. Whenever a leaf page is created in the spt, the corresponding bit is set to 1; values are never reset. So even if entries are zapped, we still knows which pages were ever mapped, a conservative estimate of which pages were ever modified. The bitmap has to be kept for the lifetime of the VM, but the 2MB granularity means the size is small: even a 16TB VM would only have a bitmap size of 1MB.

Note that for a VMM to properly support the hypercall, it also has to track all the pages it itself touches on behalf of the VM; this is clearly beyond the scope of this patch, but it is on the VMM to merge these two sets of information in some way.

---
 Documentation/virt/kvm/api.rst                |  67 ++++++
 arch/x86/include/asm/kvm_host.h               |   9 +
 arch/x86/kvm/mmu/ever_mapped.h                |  33 +++
 arch/x86/kvm/mmu/tdp_mmu.c                    |  12 ++
 arch/x86/kvm/x86.c                            | 110 ++++++++++
 include/uapi/linux/kvm.h                      |  15 ++
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../testing/selftests/kvm/ever_mapped_test.c  | 191 ++++++++++++++++++
 8 files changed, 438 insertions(+)
 create mode 100644 arch/x86/kvm/mmu/ever_mapped.h
 create mode 100644 tools/testing/selftests/kvm/ever_mapped_test.c
Documentation/virt/kvm/api.rst +67 -0
index a5f9ee92f43e..ce6e1d15d48d 100644
@@ -6566,6 +6566,53 @@ KVM_S390_KEYOP_SSKE
   Sets the storage key for the guest address ``guest_addr`` to the key
   specified in ``key``, returning the previous value in ``key``.
 
+4.145 KVM_GET_EVER_MAPPED_LOG
+-----------------------------
+
+:Capability: KVM_CAP_EVER_MAPPED
+:Architectures: x86
+:Type: vm ioctl
+:Parameters: struct kvm_ever_mapped_log
+:Returns: 0 on success, -ERRNO on error
+
+Errors:
+
+  ======     ============================================================
+  ENOENT     KVM_CAP_EVER_MAPPED has not been enabled for this VM
+  EINVAL     ``granule_shift`` is below 21 (internal tracking granularity) or
+             above 63
+  EINVAL     ``first_granule`` + ``num_granules`` extendends beyond the maxgpa
+             set up when KVM_CAP_EVER_MAPPED was enabled
+  EINVAL     ``flags`` is not 0
+  EFAULT     copy_{from,to}_user of ``bitmap`` failed, e.g., bitmap wasn't
+             sufficiently sized
+  ENOMEM     setting up temporary bitmap for copying to userspace failed
+  ======     ============================================================
+
+This requires KVM_CAP_EVER_MAPPED to have been enabled for the VM with a maxgpa
+parameter.
+
+::
+
+  struct kvm_ever_mapped_log {
+	__u64 first_granule;
+	__u64 num_granules;
+	__u32 granule_shift;
+	__u32 flags;
+	union {
+		void __user *bitmap;
+		__u64 padding;
+	};
+	__u64 reserved[4];
+  };
+
+``first_granule``, ``num_granules``, and ``granule_shift`` encode the GPA range
+the caller inquires about. For example, values of 2, 25, and 21, respectively,
+inquire about the address range [4MB, 54MB). The caller provides a ``bitmap``
+of the required length (in the above case 4 bytes). The call will write a 1 if
+any memory in the granule has been mapped at some point since
+KVM_CAP_EVER_MAPPED was enabled, 0 if not.
+
 .. _kvm_run:
 
 5. The kvm_run structure
@@ -8949,6 +8996,26 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key
 interpretation are disabled. If cmma has already been enabled or the
 hpage_2g module parameter is not set to 1, -EINVAL is returned.
 
+7.48 KVM_CAP_EVER_MAPPED
+------------------------
+
+:Architectures: x86
+:Parameters: args[0] - maximum GPA to track mapped memory
+:Returns: 0 on success, -EOPNOTSUPP if TDP is not available, -EINVAL if args[0]
+          is 0 or beyond the maximum possible GPA, -ENOMEM if tracking setup
+          failed, -EEXIST if already set up.
+
+The presence of this capability indicates that KVM can track if pages have ever
+been mapped by the guest. This allows a conservative estimate of which pages
+are not zero any more.
+
+On enabling the capability, the caller provides a max GPA value. KVM will then
+track GPAs between 0 and this value. Note that this capability can be enabled
+at any time, but it is strongly recommended to enable it before vCPUs start
+running. Any page accesses by the guest before capability enablement will not
+be tracked. The max GPA value can only be set once; subsequent attempts to
+enable the capability with a different or even same value will fail.
+
 8. Other capabilities.
 ======================
 
arch/x86/include/asm/kvm_host.h +9 -0
index b517257a6315..c607a9193f40 100644
@@ -1533,6 +1533,15 @@ struct kvm_arch {
 	 */
 	bool shadow_root_allocated;
 
+	/*
+	 * Tracks which guest pages have ever been mapped. This allows
+	 * tracking which memory is guaranteed to still be zero, e.g., for
+	 * supporting the HvExtCallGetBootZeroedMemory enlightenment.
+	 * The granularity is defined by KVM_EVER_MAPPED_SHIFT.
+	 */
+	unsigned long *ever_mapped_bitmap;
+	u64 ever_mapped_max_gpa;
+
 #ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING
 	/*
 	 * If set, the VM has (or had) an external write tracking user, and
arch/x86/kvm/mmu/ever_mapped.h new file +33 -0
new file mode 100644
index 000000000000..f795337e6848
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_EVER_MAPPED_H
+#define __KVM_X86_EVER_MAPPED_H
+
+#include <linux/kvm_host.h>
+
+/*
+ * Track if pages have ever been mapped at an internal granularity of 2 MB.
+ */
+#define KVM_EVER_MAPPED_SHIFT 21
+
+static inline void kvm_ever_mapped_set_range(struct kvm *kvm, gfn_t gfn,
+					     u64 npages)
+{
+	const unsigned int shift = KVM_EVER_MAPPED_SHIFT - PAGE_SHIFT;
+	unsigned long start = gfn >> shift;
+	unsigned long end = (gfn + npages + (1UL << shift) - 1) >> shift;
+	unsigned long i;
+
+	end = min(end, kvm->arch.ever_mapped_max_gpa >> KVM_EVER_MAPPED_SHIFT);
+	for (i = start; i < end; i++) {
+		/*
+		 * re-mapping zapped pages and breaking up of huge pages can
+		 * trigger a number of sets on already set bits, so test first
+		 * before atomic-set. Since bits can never be unset again, this
+		 * is safe against races.
+		 */
+		if (!test_bit(i, kvm->arch.ever_mapped_bitmap))
+			set_bit(i, kvm->arch.ever_mapped_bitmap);
+	}
+}
+
+#endif /* __KVM_X86_EVER_MAPPED_H */
arch/x86/kvm/mmu/tdp_mmu.c +12 -0
index ce3f2efadb05..ea2a37a04b2f 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: GPL-2.0
 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
 
+#include "ever_mapped.h"
 #include "mmu.h"
 #include "mmu_internal.h"
 #include "mmutrace.h"
@@ -554,6 +555,17 @@ static int __handle_changed_spte(struct kvm *kvm, struct kvm_mmu_page *sp,
 		return 0;
 	}
 
+	/*
+	 * Whenever a page becomes a fresh leaf, mark the memory as mapped.
+	 * On hugepage breakup, this triggers for each new lower-level page,
+	 * even though the memory was already marked when the hugepage was
+	 * mapped, but the overhead is negligible.
+	 */
+	if (unlikely(kvm->arch.ever_mapped_bitmap) &&
+	    is_leaf && !was_leaf) {
+		kvm_ever_mapped_set_range(kvm, gfn, KVM_PAGES_PER_HPAGE(level));
+	}
+
 	/*
 	 * Recursively handle child PTs if the change removed a subtree from
 	 * the paging structure.  Note the WARN on the PFN changing without the
arch/x86/kvm/x86.c +110 -0

140 lines not on this page · raw message

include/uapi/linux/kvm.h +15 -0
index 419011097fa8..d0be94946a77 100644
@@ -997,6 +997,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_KEYOP 247
 #define KVM_CAP_S390_VSIE_ESAMODE 248
 #define KVM_CAP_S390_HPAGE_2G 249
+#define KVM_CAP_EVER_MAPPED 250
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1670,4 +1671,18 @@ struct kvm_pre_fault_memory {
 	__u64 padding[5];
 };
 
+#define KVM_GET_EVER_MAPPED_LOG  _IOW(KVMIO,  0xd6, struct kvm_ever_mapped_log)
+
+struct kvm_ever_mapped_log {
+	__u64 first_granule;
+	__u64 num_granules;
+	__u32 granule_shift;
+	__u32 flags;
+	union {
+		void __user *bitmap;
+		__u64 padding;
+	};
+	__u64 reserved[4];
+};
+
 #endif /* __LINUX_KVM_H */
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 4ace12606e93..1eb0bac1920d 100644
@@ -162,6 +162,7 @@ TEST_GEN_PROGS_x86 += rseq_test
 TEST_GEN_PROGS_x86 += steal_time
 TEST_GEN_PROGS_x86 += system_counter_offset_test
 TEST_GEN_PROGS_x86 += pre_fault_memory_test
+TEST_GEN_PROGS_x86 += ever_mapped_test
 
 # Compiled outputs used by test targets
 TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
tools/testing/selftests/kvm/ever_mapped_test.c new file +191 -0

191 lines not on this page · raw message

Signature · 1 line
2.54.0
#1 lore raw
Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider: - [High] The `ever_mapped_bitmap` is dynamically allocated during VM capability enablement but never freed…

Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [High] The ever_mapped_bitmap is dynamically allocated during VM capability enablement but never freed during VM teardown.
- [High] Buffer overflow and KASAN out-of-bounds access due to a byte-aligned allocation size being manipulated by unsigned long bit operations.
- [High] Missing cond_resched() in the fallback iteration loop allows a single ioctl to monopolize the CPU, causing soft lockups.
- [High] KVM_CAP_EVER_MAPPED enablement succeeds based on the global tdp_enabled hardware flag, but silently fails to track memory if the legacy shadow MMU is actively handling the VM.
- [Medium] Unvalidated reserved fields in struct kvm_ever_mapped_log break future ABI compatibility.
- [Medium] Lockless reading of kvm->arch.ever_mapped_max_gpa and ever_mapped_bitmap can cause TOCTOU races and spurious errors.
--

--- Patch [1]: [PATCH] kvm: add EVER_MAPPED ioctl and capability --- commit bb0c368d3b83d1838e5cca827b1430b4b426ca4d
Author: Florian Schmidt <[email protected]>
Subject: kvm: add EVER_MAPPED ioctl and capability

This commit introduces a new capability, KVM_CAP_EVER_MAPPED, which allows a VMM to track which guest pages have ever been mapped using a bitmap. It also adds an ioctl, KVM_GET_EVER_MAPPED_LOG, to retrieve this tracking information.

Quoted 4 lines
arch/x86/kvm/x86.c +0 -0
index 0626e835e9eb7..89cbcf8399f49 100644

[ ... ]

arch/x86/kvm/x86.c int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) · line 2397

Quoted 7 lines
arch/x86/kvm/x86.c +3 -0
@@ -2391,6 +2392,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_READONLY_MEM:
 		r = kvm ? kvm_arch_has_readonly_mem(kvm) : 1;
 		break;
+	case KVM_CAP_EVER_MAPPED:
+		r = (tdp_enabled && IS_ENABLED(CONFIG_X86_64));
+		break;

[Severity: High]
Does this successfully track memory when the VM is running under the legacy shadow MMU?

If a VM uses the legacy shadow MMU (for example, with nested L2 guests or the kvm.tdp_mmu=0 module parameter), the page mappings are handled in mmu.c, which bypasses the kvm_ever_mapped_set_range() hook added to tdp_mmu.c. Since this check only verifies the global tdp_enabled hardware flag, could this silently fail to track memory and cause userspace to observe a clean log when the pages were actually modified?

[ ... ]

Quoted 12 lines
arch/x86/kvm/x86.c +8 -0
@@ -4190,6 +4194,36 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		mutex_unlock(&kvm->lock);
 		break;
 	}
+	case KVM_CAP_EVER_MAPPED: {
+		unsigned long *mapped_bitmap;
+		unsigned long num_granules;
+
+		r = 0;
+		mutex_lock(&kvm->lock);
+		if (!tdp_enabled) {
+			r = -EOPNOTSUPP;

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4214

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+		} else if (kvm->arch.ever_mapped_bitmap) {
+			r = -EEXIST;
+		} else if (cap->args[0] == 0 ||
+			   cap->args[0] > (1ULL << kvm_host.maxphyaddr)) {
+			r = -EINVAL;
+		} else {
+			num_granules = DIV_ROUND_UP(cap->args[0], 1ULL << KVM_EVER_MAPPED_SHIFT);
+			mapped_bitmap =
+				kvcalloc(BITS_TO_LONGS(num_granules),
+					 sizeof(long), GFP_KERNEL_ACCOUNT);

[Severity: High]
Is this allocated memory ever freed?

When a VM is destroyed via the kvm_arch_destroy_vm() path, does it clean up the ever_mapped_bitmap? An unprivileged process could repeatedly create VMs, enable this capability, and destroy the VMs, potentially exhausting host memory if the bitmap isn't explicitly freed.

Quoted 12 lines
arch/x86/kvm/x86.c +12 -0
+			if (!mapped_bitmap)
+				r = -ENOMEM;
+			else {
+				write_lock(&kvm->mmu_lock);
+				kvm->arch.ever_mapped_bitmap = mapped_bitmap;
+				kvm->arch.ever_mapped_max_gpa = num_granules << KVM_EVER_MAPPED_SHIFT;
+				write_unlock(&kvm->mmu_lock);
+			}
+		}
+		mutex_unlock(&kvm->lock);
+		break;
+	}

[ ... ]

Quoted 7 lines
arch/x86/kvm/x86.c +3 -0
@@ -4197,6 +4231,73 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 	return r;
 }
 
+static int kvm_vm_ioctl_get_ever_mapped_log(struct kvm *kvm,
+					    struct kvm_ever_mapped_log *log)
+{

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4246

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+	u64 max_granules;
+	unsigned int coarseness;
+	unsigned long *bitmap;
+	int r;
+
+	if (!kvm->arch.ever_mapped_bitmap)
+		return -ENOENT;
+
+	if (log->flags)
+		return -EINVAL;

[Severity: Medium]
Should we also validate that the reserved fields in the log structure are strictly zero here?

If userspace leaves log->reserved uninitialized, stack garbage could be passed in. Does this prevent safely repurposing these fields in the future without breaking ABI compatibility with older binaries?

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4251

Quoted 5 lines
arch/x86/kvm/x86.c +5 -0
+
+	if (log->granule_shift < KVM_EVER_MAPPED_SHIFT || log->granule_shift >= 64)
+		return -EINVAL;
+
+	max_granules = kvm->arch.ever_mapped_max_gpa >> log->granule_shift;

[Severity: Medium]
Could this lockless read of ever_mapped_max_gpa cause a TOCTOU race?

If this is called concurrently with KVM_CAP_EVER_MAPPED enablement, could the reader observe a non-NULL ever_mapped_bitmap but a stale max_gpa of 0 due to compiler reordering or missing memory barriers?

If so, max_granules would evaluate to 0 and the ioctl might erroneously fail with -EINVAL.

Quoted 14 lines
arch/x86/kvm/x86.c +14 -0
+	if (log->num_granules > max_granules ||
+	    log->first_granule > max_granules - log->num_granules)
+		return -EINVAL;
+
+	/*
+	 * For the case where the request uses the same granularity as our internal
+	 * tracking, and we are byte-aligned (the expected common case), we can
+	 * just copy_to_user. Otherwise, we have to create a new bitmap to pass.
+	 *
+	 * We do not lock against concurrent vCPUs mapping new memory on any of
+	 * these operations. Bits are set atomically and never cleared, so any race
+	 * here is indistinguishable from a write happening after this handler
+	 * finishes but before the caller reads the results.
+	 */

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4275

Quoted 10 lines
arch/x86/kvm/x86.c +10 -0
+	coarseness = log->granule_shift - KVM_EVER_MAPPED_SHIFT;
+	if (coarseness == 0 && !(log->first_granule & 7) && !(log->num_granules & 7)) {
+		if (copy_to_user(log->bitmap,
+				 (u8 *)kvm->arch.ever_mapped_bitmap + log->first_granule / 8,
+				 log->num_granules / 8))
+			return -EFAULT;
+		return 0;
+	}
+
+	bitmap = kvzalloc(DIV_ROUND_UP(log->num_granules, 8), GFP_KERNEL_ACCOUNT);

[Severity: High]
Does this allocation size safely accommodate the unsigned long bit operations that follow?

The buffer is allocated with exact byte granularity here using kvzalloc(DIV_ROUND_UP(log->num_granules, 8), ...), but it is later populated using __set_bit() which operates on 8-byte (unsigned long) boundaries.

If the requested size is not a multiple of 8 bytes (for example, 9 bytes for 65 granules), could __set_bit() access memory past the requested allocation boundary and trigger a KASAN out-of-bounds splat?

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4281

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+	if (!bitmap)
+		return -ENOMEM;
+
+	for (u64 gran = log->first_granule;
+	     gran < log->first_granule + log->num_granules;
+	     gran++) {

[Severity: High]
Can this loop monopolize the CPU and trigger a soft lockup?

If userspace calls this ioctl with unaligned parameters that bypass the copy_to_user fast path, log->num_granules can be very large (up to 2^31 for a 52-bit physical address space and a 2MB granule).

Does iterating over a potentially massive user-controlled scale without a cond_resched() in this loop risk keeping the CPU occupied long enough to trigger the soft lockup watchdog?

Quoted 6 lines
arch/x86/kvm/x86.c +6 -0
+		unsigned long start = gran << coarseness;
+		unsigned long end = start + (1UL << coarseness);
+
+		if (find_next_bit(kvm->arch.ever_mapped_bitmap, end, start) < end)
+			__set_bit(gran - log->first_granule, bitmap);
+	}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1