Skip to contentSkip to conversation

KVM: Mapping guest_memfd backed memory at the host for software protected VMs

PATCH Superseded Fuad Tabba 10 participants 64 messages Apr 30, 2025 – May 12, 2025

Subsystems (9)

v8 of 16 Superseded Latest is v16 →

lore
Share & tools
lore

Fuad Tabba

author
#0 lore raw

v8 cover KVM: Mapping guest_memfd backed memory at the host for software protected VMs

Main changes since v7 [1]:
- Renaming/refactoring to decouple guest memory from whether the underlying memory is private vs being backed by guest_memfd
- Drop folio_put() callback patches
- Fixes based on feedback from the previous series
- Rebase on Linux 6.15-rc4

The purpose of this series is to allow mapping guest_memfd backed memory at the host. This support enables VMMs like Firecracker to run VM guests backed completely by guest_memfd [2]. Combined with Patrick's series for direct map removal in guest_memfd [3], this would allow running VMs that offer additional hardening against Spectre-like transient execution attacks.

This series will also serve as a base for _restricted_ mmap() support for guest_memfd backed memory at the host for CoCos that allow sharing guest memory in-place with the host [4].

Patches 1 to 7 are mainly about decoupling the concept of guest memory being private vs guest memory being backed by guest_memfd. They are mostly refactoring and renaming.

Patch 8 adds support for in-place shared memory, as well as the ability to map it by the host as long as it is shared, gated by a new configuration option, and adviertised to userspace by a new capability.

Patches 9 to 12 add arm64 and x86 support for in-place shared memory.

Patch 13 expands the guest_memfd selftest to test in-place shared memory when avaialble.

To test this patch series on x86 (I use a standard Debian image):

Build:

- Build the kernel with the following config options enabled:
defconfigs:

	x86_64_defconfig
	kvm_guest.config

Additional config options to enable:
KVM_SW_PROTECTED_VM
KVM_GMEM_SHARED_MEM

- Build the kernel kvm selftest tools/testing/selftests/kvm, you only need guest_memfd_test, e.g.:
make EXTRA_CFLAGS="-static -DDEBUG" -C tools/testing/selftests/kvm

- Build kvmtool [5] lkvm-static (I build it on a different machine). make lkvm-static

Run:
Boot your Linux image with the kernel you built above.

The selftest you can run as it is:
./guest_memfd_test

For kvmtool, where bzImage is the same as the host's:

	./lkvm-static run -c 2 -m 512 -p "break=mount" --kernel bzImage --debug --guest_memfd --sw_protected

To test this patch series on arm64 (I use a standard Debian image):

Build:

- Build the kernel with defconfig

- Build the kernel kvm selftest tools/testing/selftests/kvm, you only need guest_memfd_test.

- Build kvmtool [5] lkvm-static (I cross compile it on a different machine). You are likely to need libfdt as well.

For libfdt (in the same directory as kvmtool):

	git clone git://git.kernel.org/pub/scm/utils/dtc/dtc.git
	cd dtc
	export CC=aarch64-linux-gnu-gcc
	make
	cd ..

Then for kvmtool:
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- LIBFDT_DIR=./dtc/libfdt/ lkvm-static

Run:
Boot your Linux image with the kernel you built above.

The selftest you can run as it is:
./guest_memfd_test

For kvmtool, where Image is the same as the host's, and rootfs is your rootfs image (in case kvmtool can't figure it out):

	./lkvm-static run -c 2 -m 512 -d rootfs --kernel Image --force-pci --irqchip gicv3 --debug --guest_memfd --sw_protected

You can find (potentially slightly outdated) instructions on how to a full arm64 system stack under QEMU here [6].

Cheers,
/fuad

[1] https://lore.kernel.org/all/[email protected]/
[2] https://github.com/firecracker-microvm/firecracker/tree/feature/secret-hiding
[3] https://lore.kernel.org/all/[email protected]/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://android-kvm.googlesource.com/kvmtool/+/refs/heads/tabba/guestmem-basic-6.15
[6] https://mirrors.edge.kernel.org/pub/linux/kernel/people/will/docs/qemu/qemu-arm64-howto.html

Fuad Tabba (13):
KVM: Rename CONFIG_KVM_PRIVATE_MEM to CONFIG_KVM_GMEM
KVM: Rename CONFIG_KVM_GENERIC_PRIVATE_MEM to
CONFIG_KVM_GENERIC_GMEM_POPULATE
KVM: Rename kvm_arch_has_private_mem() to kvm_arch_supports_gmem()
KVM: x86: Rename kvm->arch.has_private_mem to kvm->arch.supports_gmem
KVM: Rename kvm_slot_can_be_private() to kvm_slot_has_gmem()
KVM: x86: Generalize private fault lookups to guest_memfd fault
lookups
KVM: Fix comments that refer to slots_lock
KVM: guest_memfd: Allow host to map guest_memfd() pages
KVM: arm64: Refactor user_mem_abort() calculation of force_pte
KVM: arm64: Handle guest_memfd()-backed guest page faults
KVM: arm64: Enable mapping guest_memfd in arm64
KVM: x86: KVM_X86_SW_PROTECTED_VM to support guest_memfd shared memory
KVM: guest_memfd: selftests: guest_memfd mmap() test when mapping is
allowed

 arch/arm64/include/asm/kvm_host.h             | 12 +++
 arch/arm64/kvm/Kconfig                        |  1 +
 arch/arm64/kvm/mmu.c                          | 76 +++++++++------
 arch/x86/include/asm/kvm_host.h               | 17 ++--
 arch/x86/kvm/Kconfig                          |  4 +-
 arch/x86/kvm/mmu/mmu.c                        | 31 +++---
 arch/x86/kvm/svm/sev.c                        |  4 +-
 arch/x86/kvm/svm/svm.c                        |  4 +-
 arch/x86/kvm/x86.c                            |  3 +-
 include/linux/kvm_host.h                      | 44 +++++++--
 include/uapi/linux/kvm.h                      |  1 +
 tools/testing/selftests/kvm/Makefile.kvm      |  1 +
 .../testing/selftests/kvm/guest_memfd_test.c  | 75 +++++++++++++--
 virt/kvm/Kconfig                              | 15 ++-
 virt/kvm/Makefile.kvm                         |  2 +-
 virt/kvm/guest_memfd.c                        | 96 ++++++++++++++++++-
 virt/kvm/kvm_main.c                           | 21 ++--
 virt/kvm/kvm_mm.h                             |  4 +-
 18 files changed, 316 insertions(+), 95 deletions(-)

base-commit: b4432656b36e5cc1d50a1f2dc15357543add530e

Signature · 1 line
2.49.0.901.g37484f566f-goog
author
#1 lore raw

PATCH v8 1/13 KVM: Rename CONFIG_KVM_PRIVATE_MEM to CONFIG_KVM_GMEM

The option KVM_PRIVATE_MEM enables guest_memfd in general. Subsequent patches add shared memory support to guest_memfd. Therefore, rename it to KVM_GMEM to make its purpose clearer.

---
 arch/x86/include/asm/kvm_host.h |  2 +-
 include/linux/kvm_host.h        | 10 +++++-----
 virt/kvm/Kconfig                |  8 ++++----
 virt/kvm/Makefile.kvm           |  2 +-
 virt/kvm/kvm_main.c             |  4 ++--
 virt/kvm/kvm_mm.h               |  4 ++--
 6 files changed, 15 insertions(+), 15 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
index 7bc174a1f1cb..52f6f6d08558 100644
@@ -2253,7 +2253,7 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level,
 		       int tdp_max_root_level, int tdp_huge_page_level);
 
 
-#ifdef CONFIG_KVM_PRIVATE_MEM
+#ifdef CONFIG_KVM_GMEM
 #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
 #else
 #define kvm_arch_has_private_mem(kvm) false
include/linux/kvm_host.h +5 -5
index 291d49b9bf05..d6900995725d 100644
@@ -601,7 +601,7 @@ struct kvm_memory_slot {
 	short id;
 	u16 as_id;
 
-#ifdef CONFIG_KVM_PRIVATE_MEM
+#ifdef CONFIG_KVM_GMEM
 	struct {
 		/*
 		 * Writes protected by kvm->slots_lock.  Acquiring a
@@ -722,7 +722,7 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
  * Arch code must define kvm_arch_has_private_mem if support for private memory
  * is enabled.
  */
-#if !defined(kvm_arch_has_private_mem) && !IS_ENABLED(CONFIG_KVM_PRIVATE_MEM)
+#if !defined(kvm_arch_has_private_mem) && !IS_ENABLED(CONFIG_KVM_GMEM)
 static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
 {
 	return false;
@@ -2504,7 +2504,7 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
-	return IS_ENABLED(CONFIG_KVM_PRIVATE_MEM) &&
+	return IS_ENABLED(CONFIG_KVM_GMEM) &&
 	       kvm_get_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 }
 #else
@@ -2514,7 +2514,7 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 }
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
-#ifdef CONFIG_KVM_PRIVATE_MEM
+#ifdef CONFIG_KVM_GMEM
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
 		     int *max_order);
@@ -2527,7 +2527,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 	KVM_BUG_ON(1, kvm);
 	return -EIO;
 }
-#endif /* CONFIG_KVM_PRIVATE_MEM */
+#endif /* CONFIG_KVM_GMEM */
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
virt/kvm/Kconfig +4 -4
index 727b542074e7..49df4e32bff7 100644
@@ -112,19 +112,19 @@ config KVM_GENERIC_MEMORY_ATTRIBUTES
        depends on KVM_GENERIC_MMU_NOTIFIER
        bool
 
-config KVM_PRIVATE_MEM
+config KVM_GMEM
        select XARRAY_MULTI
        bool
 
 config KVM_GENERIC_PRIVATE_MEM
        select KVM_GENERIC_MEMORY_ATTRIBUTES
-       select KVM_PRIVATE_MEM
+       select KVM_GMEM
        bool
 
 config HAVE_KVM_ARCH_GMEM_PREPARE
        bool
-       depends on KVM_PRIVATE_MEM
+       depends on KVM_GMEM
 
 config HAVE_KVM_ARCH_GMEM_INVALIDATE
        bool
-       depends on KVM_PRIVATE_MEM
+       depends on KVM_GMEM
virt/kvm/Makefile.kvm +1 -1
index 724c89af78af..8d00918d4c8b 100644
@@ -12,4 +12,4 @@ kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o
 kvm-$(CONFIG_HAVE_KVM_IRQ_ROUTING) += $(KVM)/irqchip.o
 kvm-$(CONFIG_HAVE_KVM_DIRTY_RING) += $(KVM)/dirty_ring.o
 kvm-$(CONFIG_HAVE_KVM_PFNCACHE) += $(KVM)/pfncache.o
-kvm-$(CONFIG_KVM_PRIVATE_MEM) += $(KVM)/guest_memfd.o
+kvm-$(CONFIG_KVM_GMEM) += $(KVM)/guest_memfd.o
virt/kvm/kvm_main.c +2 -2
index e85b33a92624..4996cac41a8f 100644
@@ -4842,7 +4842,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 	case KVM_CAP_MEMORY_ATTRIBUTES:
 		return kvm_supported_mem_attributes(kvm);
 #endif
-#ifdef CONFIG_KVM_PRIVATE_MEM
+#ifdef CONFIG_KVM_GMEM
 	case KVM_CAP_GUEST_MEMFD:
 		return !kvm || kvm_arch_has_private_mem(kvm);
 #endif
@@ -5276,7 +5276,7 @@ static long kvm_vm_ioctl(struct file *filp,
 	case KVM_GET_STATS_FD:
 		r = kvm_vm_ioctl_get_stats_fd(kvm);
 		break;
-#ifdef CONFIG_KVM_PRIVATE_MEM
+#ifdef CONFIG_KVM_GMEM
 	case KVM_CREATE_GUEST_MEMFD: {
 		struct kvm_create_guest_memfd guest_memfd;
 
virt/kvm/kvm_mm.h +2 -2
index acef3f5c582a..ec311c0d6718 100644
@@ -67,7 +67,7 @@ static inline void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm,
 }
 #endif /* HAVE_KVM_PFNCACHE */
 
-#ifdef CONFIG_KVM_PRIVATE_MEM
+#ifdef CONFIG_KVM_GMEM
 void kvm_gmem_init(struct module *module);
 int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args);
 int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
@@ -91,6 +91,6 @@ static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot)
 {
 	WARN_ON_ONCE(1);
 }
-#endif /* CONFIG_KVM_PRIVATE_MEM */
+#endif /* CONFIG_KVM_GMEM */
 
 #endif /* __KVM_MM_H__ */
Signature · 1 line
2.49.0.901.g37484f566f-goog
Reviewed-by on v8 1/13
#2 lore raw

Re: [PATCH v8 01/13] KVM: Rename CONFIG_KVM_PRIVATE_MEM to CONFIG_KVM_GMEM

Quoted 7 lines · Fuad Tabba

The option KVM_PRIVATE_MEM enables guest_memfd in general. Subsequent patches add shared memory support to guest_memfd. Therefore, rename it to KVM_GMEM to make its purpose clearer.

author
#3 lore raw

PATCH v8 2/13 KVM: Rename CONFIG_KVM_GENERIC_PRIVATE_MEM to CONFIG_KVM_GENERIC_GMEM_POPULATE

In reply to Fuad Tabba

The option KVM_GENERIC_PRIVATE_MEM enables populating a GPA range with guest data. Rename it to KVM_GENERIC_GMEM_POPULATE to make its purpose clearer.

---
 arch/x86/kvm/Kconfig     | 4 ++--
 include/linux/kvm_host.h | 2 +-
 virt/kvm/Kconfig         | 2 +-
 virt/kvm/guest_memfd.c   | 2 +-
 4 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/Kconfig +2 -2
index fe8ea8c097de..b37258253543 100644
@@ -46,7 +46,7 @@ config KVM_X86
 	select HAVE_KVM_PM_NOTIFIER if PM
 	select KVM_GENERIC_HARDWARE_ENABLING
 	select KVM_GENERIC_PRE_FAULT_MEMORY
-	select KVM_GENERIC_PRIVATE_MEM if KVM_SW_PROTECTED_VM
+	select KVM_GENERIC_GMEM_POPULATE if KVM_SW_PROTECTED_VM
 	select KVM_WERROR if WERROR
 
 config KVM
@@ -145,7 +145,7 @@ config KVM_AMD_SEV
 	depends on KVM_AMD && X86_64
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
-	select KVM_GENERIC_PRIVATE_MEM
+	select KVM_GENERIC_GMEM_POPULATE
 	select HAVE_KVM_ARCH_GMEM_PREPARE
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
 	help
include/linux/kvm_host.h +1 -1
index d6900995725d..7ca23837fa52 100644
@@ -2533,7 +2533,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
 #endif
 
-#ifdef CONFIG_KVM_GENERIC_PRIVATE_MEM
+#ifdef CONFIG_KVM_GENERIC_GMEM_POPULATE
 /**
  * kvm_gmem_populate() - Populate/prepare a GPA range with guest data
  *
virt/kvm/Kconfig +1 -1
index 49df4e32bff7..559c93ad90be 100644
@@ -116,7 +116,7 @@ config KVM_GMEM
        select XARRAY_MULTI
        bool
 
-config KVM_GENERIC_PRIVATE_MEM
+config KVM_GENERIC_GMEM_POPULATE
        select KVM_GENERIC_MEMORY_ATTRIBUTES
        select KVM_GMEM
        bool
virt/kvm/guest_memfd.c +1 -1
index b2aa6bf24d3a..befea51bbc75 100644
@@ -638,7 +638,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 }
 EXPORT_SYMBOL_GPL(kvm_gmem_get_pfn);
 
-#ifdef CONFIG_KVM_GENERIC_PRIVATE_MEM
+#ifdef CONFIG_KVM_GENERIC_GMEM_POPULATE
 long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long npages,
 		       kvm_gmem_populate_cb post_populate, void *opaque)
 {
Signature · 1 line
2.49.0.901.g37484f566f-goog
#4 lore raw

Re: [PATCH v8 02/13] KVM: Rename CONFIG_KVM_GENERIC_PRIVATE_MEM to CONFIG_KVM_GENERIC_GMEM_POPULATE

Quoted 3 lines · Fuad Tabba

The option KVM_GENERIC_PRIVATE_MEM enables populating a GPA range with guest data. Rename it to KVM_GENERIC_GMEM_POPULATE to make its purpose clearer.

I'm curious what generic means in this name?

FWICT if we are going to change the name, KVM_GMEM_POPULATE is a better name.

Ira

#5 lore raw

Re: [PATCH v8 02/13] KVM: Rename CONFIG_KVM_GENERIC_PRIVATE_MEM to CONFIG_KVM_GENERIC_GMEM_POPULATE

Quoted 6 lines · Ira Weiny
Quoted 3 lines · Fuad Tabba

The option KVM_GENERIC_PRIVATE_MEM enables populating a GPA range with guest data. Rename it to KVM_GENERIC_GMEM_POPULATE to make its purpose clearer.

I'm curious what generic means in this name?

That an architecture wants to use the generic version and not provide it's own alternative implementation.

We frequently use that term in this context, see GENERIC_IOREMAP as one example.

Signature · 3 lines
Cheers,

David / dhildenb
#6 lore raw

Re: [PATCH v8 02/13] KVM: Rename CONFIG_KVM_GENERIC_PRIVATE_MEM to CONFIG_KVM_GENERIC_GMEM_POPULATE

Quoted 6 lines · David Hildenbrand
Quoted 5 lines · Ira Weiny
Quoted 3 lines · Fuad Tabba

The option KVM_GENERIC_PRIVATE_MEM enables populating a GPA range with guest data. Rename it to KVM_GENERIC_GMEM_POPULATE to make its purpose clearer.

Quoted 7 lines
Quoted 1 line

I'm curious what generic means in this name?

That an architecture wants to use the generic version and not provide it's own alternative implementation.

We frequently use that term in this context, see GENERIC_IOREMAP as one example.

Ah ok. Thanks.

Ira

Quoted 6 lines
Signature · 4 lines
-- 
Cheers,

David / dhildenb
author
#7 lore raw

PATCH v8 3/13 KVM: Rename kvm_arch_has_private_mem() to kvm_arch_supports_gmem()

In reply to Fuad Tabba

The function kvm_arch_has_private_mem() is used to indicate whether guest_memfd is supported by the architecture, which until now implies that its private. To decouple guest_memfd support from whether the memory is private, rename this function to kvm_arch_supports_gmem().

---
 arch/x86/include/asm/kvm_host.h | 8 ++++----
 arch/x86/kvm/mmu/mmu.c          | 8 ++++----
 include/linux/kvm_host.h        | 6 +++---
 virt/kvm/kvm_main.c             | 6 +++---
 4 files changed, 14 insertions(+), 14 deletions(-)
arch/x86/include/asm/kvm_host.h +4 -4
index 52f6f6d08558..4a83fbae7056 100644
@@ -2254,9 +2254,9 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level,
 
 
 #ifdef CONFIG_KVM_GMEM
-#define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
+#define kvm_arch_supports_gmem(kvm) ((kvm)->arch.has_private_mem)
 #else
-#define kvm_arch_has_private_mem(kvm) false
+#define kvm_arch_supports_gmem(kvm) false
 #endif
 
 #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state)
@@ -2309,8 +2309,8 @@ enum {
 #define HF_SMM_INSIDE_NMI_MASK	(1 << 2)
 
 # define KVM_MAX_NR_ADDRESS_SPACES	2
-/* SMM is currently unsupported for guests with private memory. */
-# define kvm_arch_nr_memslot_as_ids(kvm) (kvm_arch_has_private_mem(kvm) ? 1 : 2)
+/* SMM is currently unsupported for guests with guest_memfd (esp private) memory. */
+# define kvm_arch_nr_memslot_as_ids(kvm) (kvm_arch_supports_gmem(kvm) ? 1 : 2)
 # define kvm_arch_vcpu_memslots_id(vcpu) ((vcpu)->arch.hflags & HF_SMM_MASK ? 1 : 0)
 # define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, (role).smm)
 #else
arch/x86/kvm/mmu/mmu.c +4 -4
index 63bb77ee1bb1..7d654506d800 100644
@@ -4917,7 +4917,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,
 	if (r)
 		return r;
 
-	if (kvm_arch_has_private_mem(vcpu->kvm) &&
+	if (kvm_arch_supports_gmem(vcpu->kvm) &&
 	    kvm_mem_is_private(vcpu->kvm, gpa_to_gfn(range->gpa)))
 		error_code |= PFERR_PRIVATE_ACCESS;
 
@@ -7683,7 +7683,7 @@ bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 	 * Zapping SPTEs in this case ensures KVM will reassess whether or not
 	 * a hugepage can be used for affected ranges.
 	 */
-	if (WARN_ON_ONCE(!kvm_arch_has_private_mem(kvm)))
+	if (WARN_ON_ONCE(!kvm_arch_supports_gmem(kvm)))
 		return false;
 
 	/* Unmap the old attribute page. */
@@ -7746,7 +7746,7 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 	 * a range that has PRIVATE GFNs, and conversely converting a range to
 	 * SHARED may now allow hugepages.
 	 */
-	if (WARN_ON_ONCE(!kvm_arch_has_private_mem(kvm)))
+	if (WARN_ON_ONCE(!kvm_arch_supports_gmem(kvm)))
 		return false;
 
 	/*
@@ -7802,7 +7802,7 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
 {
 	int level;
 
-	if (!kvm_arch_has_private_mem(kvm))
+	if (!kvm_arch_supports_gmem(kvm))
 		return;
 
 	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
include/linux/kvm_host.h +3 -3
index 7ca23837fa52..6ca7279520cf 100644
@@ -719,11 +719,11 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
 #endif
 
 /*
- * Arch code must define kvm_arch_has_private_mem if support for private memory
+ * Arch code must define kvm_arch_supports_gmem if support for guest_memfd
  * is enabled.
  */
-#if !defined(kvm_arch_has_private_mem) && !IS_ENABLED(CONFIG_KVM_GMEM)
-static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
+#if !defined(kvm_arch_supports_gmem) && !IS_ENABLED(CONFIG_KVM_GMEM)
+static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
 {
 	return false;
 }
virt/kvm/kvm_main.c +3 -3
index 4996cac41a8f..2468d50a9ed4 100644
@@ -1531,7 +1531,7 @@ static int check_memory_region_flags(struct kvm *kvm,
 {
 	u32 valid_flags = KVM_MEM_LOG_DIRTY_PAGES;
 
-	if (kvm_arch_has_private_mem(kvm))
+	if (kvm_arch_supports_gmem(kvm))
 		valid_flags |= KVM_MEM_GUEST_MEMFD;
 
 	/* Dirty logging private memory is not currently supported. */
@@ -2362,7 +2362,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 static u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
-	if (!kvm || kvm_arch_has_private_mem(kvm))
+	if (!kvm || kvm_arch_supports_gmem(kvm))
 		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
 
 	return 0;
@@ -4844,7 +4844,7 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 #endif
 #ifdef CONFIG_KVM_GMEM
 	case KVM_CAP_GUEST_MEMFD:
-		return !kvm || kvm_arch_has_private_mem(kvm);
+		return !kvm || kvm_arch_supports_gmem(kvm);
 #endif
 	default:
 		break;
Signature · 1 line
2.49.0.901.g37484f566f-goog
Reviewed-by on v8 3/13
#8 lore raw

Re: [PATCH v8 03/13] KVM: Rename kvm_arch_has_private_mem() to kvm_arch_supports_gmem()

Quoted 8 lines · Fuad Tabba

The function kvm_arch_has_private_mem() is used to indicate whether guest_memfd is supported by the architecture, which until now implies that its private. To decouple guest_memfd support from whether the memory is private, rename this function to kvm_arch_supports_gmem().

[snip]

author
#9 lore raw

PATCH v8 4/13 KVM: x86: Rename kvm->arch.has_private_mem to kvm->arch.supports_gmem

In reply to Fuad Tabba

The bool has_private_mem is used to indicate whether guest_memfd is supported. Rename it to supports_gmem to make its meaning clearer and to decouple memory being private from guest_memfd.

---
 arch/x86/include/asm/kvm_host.h | 4 ++--
 arch/x86/kvm/mmu/mmu.c          | 2 +-
 arch/x86/kvm/svm/svm.c          | 4 ++--
 arch/x86/kvm/x86.c              | 3 +--
 4 files changed, 6 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
index 4a83fbae7056..709cc2a7ba66 100644
@@ -1331,7 +1331,7 @@ struct kvm_arch {
 	unsigned int indirect_shadow_pages;
 	u8 mmu_valid_gen;
 	u8 vm_type;
-	bool has_private_mem;
+	bool supports_gmem;
 	bool has_protected_state;
 	bool pre_fault_allowed;
 	struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES];
@@ -2254,7 +2254,7 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level,
 
 
 #ifdef CONFIG_KVM_GMEM
-#define kvm_arch_supports_gmem(kvm) ((kvm)->arch.has_private_mem)
+#define kvm_arch_supports_gmem(kvm) ((kvm)->arch.supports_gmem)
 #else
 #define kvm_arch_supports_gmem(kvm) false
 #endif
arch/x86/kvm/mmu/mmu.c +1 -1
index 7d654506d800..734d71ec97ef 100644
@@ -3486,7 +3486,7 @@ static bool page_fault_can_be_fast(struct kvm *kvm, struct kvm_page_fault *fault
 	 * on RET_PF_SPURIOUS until the update completes, or an actual spurious
 	 * case might go down the slow path. Either case will resolve itself.
 	 */
-	if (kvm->arch.has_private_mem &&
+	if (kvm->arch.supports_gmem &&
 	    fault->is_private != kvm_mem_is_private(kvm, fault->gfn))
 		return false;
 
arch/x86/kvm/svm/svm.c +2 -2
index d5d0c5c3300b..b391dd6208cf 100644
@@ -5048,8 +5048,8 @@ static int svm_vm_init(struct kvm *kvm)
 			(type == KVM_X86_SEV_ES_VM || type == KVM_X86_SNP_VM);
 		to_kvm_sev_info(kvm)->need_init = true;
 
-		kvm->arch.has_private_mem = (type == KVM_X86_SNP_VM);
-		kvm->arch.pre_fault_allowed = !kvm->arch.has_private_mem;
+		kvm->arch.supports_gmem = (type == KVM_X86_SNP_VM);
+		kvm->arch.pre_fault_allowed = !kvm->arch.supports_gmem;
 	}
 
 	if (!pause_filter_count || !pause_filter_thresh)
arch/x86/kvm/x86.c +1 -2
index df5b99ea1f18..5b11ef131d5c 100644
@@ -12716,8 +12716,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 		return -EINVAL;
 
 	kvm->arch.vm_type = type;
-	kvm->arch.has_private_mem =
-		(type == KVM_X86_SW_PROTECTED_VM);
+	kvm->arch.supports_gmem = (type == KVM_X86_SW_PROTECTED_VM);
 	/* Decided by the vendor code for other VM types.  */
 	kvm->arch.pre_fault_allowed =
 		type == KVM_X86_DEFAULT_VM || type == KVM_X86_SW_PROTECTED_VM;
Signature · 1 line
2.49.0.901.g37484f566f-goog
Reviewed-by on v8 4/13
#10 lore raw

Re: [PATCH v8 04/13] KVM: x86: Rename kvm->arch.has_private_mem to kvm->arch.supports_gmem

Quoted 7 lines · Fuad Tabba

The bool has_private_mem is used to indicate whether guest_memfd is supported. Rename it to supports_gmem to make its meaning clearer and to decouple memory being private from guest_memfd.

[snip]

author
#11 lore raw

PATCH v8 5/13 KVM: Rename kvm_slot_can_be_private() to kvm_slot_has_gmem()

In reply to Fuad Tabba

The function kvm_slot_can_be_private() is used to check whether a memory slot is backed by guest_memfd. Rename it to kvm_slot_has_gmem() to make that clearer and to decouple memory being private from guest_memfd.

---
 arch/x86/kvm/mmu/mmu.c   | 4 ++--
 arch/x86/kvm/svm/sev.c   | 4 ++--
 include/linux/kvm_host.h | 2 +-
 virt/kvm/guest_memfd.c   | 2 +-
 4 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -2
index 734d71ec97ef..6d5dd869c890 100644
@@ -3283,7 +3283,7 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn)
 {
-	bool is_private = kvm_slot_can_be_private(slot) &&
+	bool is_private = kvm_slot_has_gmem(slot) &&
 			  kvm_mem_is_private(kvm, gfn);
 
 	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_private);
@@ -4496,7 +4496,7 @@ static int kvm_mmu_faultin_pfn_private(struct kvm_vcpu *vcpu,
 {
 	int max_order, r;
 
-	if (!kvm_slot_can_be_private(fault->slot)) {
+	if (!kvm_slot_has_gmem(fault->slot)) {
 		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
 		return -EFAULT;
 	}
arch/x86/kvm/svm/sev.c +2 -2
index 0bc708ee2788..fbf55821d62e 100644
@@ -2378,7 +2378,7 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp)
 	mutex_lock(&kvm->slots_lock);
 
 	memslot = gfn_to_memslot(kvm, params.gfn_start);
-	if (!kvm_slot_can_be_private(memslot)) {
+	if (!kvm_slot_has_gmem(memslot)) {
 		ret = -EINVAL;
 		goto out;
 	}
@@ -4682,7 +4682,7 @@ void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code)
 	}
 
 	slot = gfn_to_memslot(kvm, gfn);
-	if (!kvm_slot_can_be_private(slot)) {
+	if (!kvm_slot_has_gmem(slot)) {
 		pr_warn_ratelimited("SEV: Unexpected RMP fault, non-private slot for GPA 0x%llx\n",
 				    gpa);
 		return;
include/linux/kvm_host.h +1 -1
index 6ca7279520cf..d9616ee6acc7 100644
@@ -614,7 +614,7 @@ struct kvm_memory_slot {
 #endif
 };
 
-static inline bool kvm_slot_can_be_private(const struct kvm_memory_slot *slot)
+static inline bool kvm_slot_has_gmem(const struct kvm_memory_slot *slot)
 {
 	return slot && (slot->flags & KVM_MEM_GUEST_MEMFD);
 }
virt/kvm/guest_memfd.c +1 -1
index befea51bbc75..6db515833f61 100644
@@ -654,7 +654,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long
 		return -EINVAL;
 
 	slot = gfn_to_memslot(kvm, start_gfn);
-	if (!kvm_slot_can_be_private(slot))
+	if (!kvm_slot_has_gmem(slot))
 		return -EINVAL;
 
 	file = kvm_gmem_get_file(slot);
Signature · 1 line
2.49.0.901.g37484f566f-goog
Reviewed-by on v8 5/13
#12 lore raw

Re: [PATCH v8 05/13] KVM: Rename kvm_slot_can_be_private() to kvm_slot_has_gmem()

Quoted 7 lines · Fuad Tabba

The function kvm_slot_can_be_private() is used to check whether a memory slot is backed by guest_memfd. Rename it to kvm_slot_has_gmem() to make that clearer and to decouple memory being private from guest_memfd.

[snip]

author
#13 lore raw

PATCH v8 6/13 KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

In reply to Fuad Tabba

Until now, faults to private memory backed by guest_memfd are always consumed from guest_memfd whereas faults to shared memory are consumed from anonymous memory. Subsequent patches will allow sharing guest_memfd backed memory in-place, and mapping it by the host. Faults to in-place shared memory should be consumed from guest_memfd as well.

In order to facilitate that, generalize the fault lookups. Currently, only private memory is consumed from guest_memfd and therefore as it stands, this patch does not change the behavior.

---
 arch/x86/kvm/mmu/mmu.c   | 19 +++++++++----------
 include/linux/kvm_host.h |  6 ++++++
 2 files changed, 15 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +9 -10
index 6d5dd869c890..08eebd24a0e1 100644
@@ -3258,7 +3258,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 				       const struct kvm_memory_slot *slot,
-				       gfn_t gfn, int max_level, bool is_private)
+				       gfn_t gfn, int max_level, bool is_gmem)
 {
 	struct kvm_lpage_info *linfo;
 	int host_level;
@@ -3270,7 +3270,7 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 			break;
 	}
 
-	if (is_private)
+	if (is_gmem)
 		return max_level;
 
 	if (max_level == PG_LEVEL_4K)
@@ -3283,10 +3283,9 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn)
 {
-	bool is_private = kvm_slot_has_gmem(slot) &&
-			  kvm_mem_is_private(kvm, gfn);
+	bool is_gmem = kvm_slot_has_gmem(slot) && kvm_mem_from_gmem(kvm, gfn);
 
-	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_private);
+	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_gmem);
 }
 
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -4465,7 +4464,7 @@ static inline u8 kvm_max_level_for_order(int order)
 	return PG_LEVEL_4K;
 }
 
-static u8 kvm_max_private_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
+static u8 kvm_max_gmem_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
 					u8 max_level, int gmem_order)
 {
 	u8 req_max_level;
@@ -4491,7 +4490,7 @@ static void kvm_mmu_finish_page_fault(struct kvm_vcpu *vcpu,
 				 r == RET_PF_RETRY, fault->map_writable);
 }
 
-static int kvm_mmu_faultin_pfn_private(struct kvm_vcpu *vcpu,
+static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 				       struct kvm_page_fault *fault)
 {
 	int max_order, r;
@@ -4509,8 +4508,8 @@ static int kvm_mmu_faultin_pfn_private(struct kvm_vcpu *vcpu,
 	}
 
 	fault->map_writable = !(fault->slot->flags & KVM_MEM_READONLY);
-	fault->max_level = kvm_max_private_mapping_level(vcpu->kvm, fault->pfn,
-							 fault->max_level, max_order);
+	fault->max_level = kvm_max_gmem_mapping_level(vcpu->kvm, fault->pfn,
+						      fault->max_level, max_order);
 
 	return RET_PF_CONTINUE;
 }
@@ -4521,7 +4520,7 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	unsigned int foll = fault->write ? FOLL_WRITE : 0;
 
 	if (fault->is_private)
-		return kvm_mmu_faultin_pfn_private(vcpu, fault);
+		return kvm_mmu_faultin_pfn_gmem(vcpu, fault);
 
 	foll |= FOLL_NOWAIT;
 	fault->pfn = __kvm_faultin_pfn(fault->slot, fault->gfn, foll,
include/linux/kvm_host.h +6 -0
index d9616ee6acc7..cdcd7ac091b5 100644
@@ -2514,6 +2514,12 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 }
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
+static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
+{
+	/* For now, only private memory gets consumed from guest_memfd. */
+	return kvm_mem_is_private(kvm, gfn);
+}
+
 #ifdef CONFIG_KVM_GMEM
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
Signature · 1 line
2.49.0.901.g37484f566f-goog
#14 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 32 lines · Fuad Tabba

Until now, faults to private memory backed by guest_memfd are always consumed from guest_memfd whereas faults to shared memory are consumed from anonymous memory. Subsequent patches will allow sharing guest_memfd backed memory in-place, and mapping it by the host. Faults to in-place shared memory should be consumed from guest_memfd as well.

In order to facilitate that, generalize the fault lookups. Currently, only private memory is consumed from guest_memfd and therefore as it stands, this patch does not change the behavior.

---
 arch/x86/kvm/mmu/mmu.c   | 19 +++++++++----------
 include/linux/kvm_host.h |  6 ++++++
 2 files changed, 15 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index 6d5dd869c890..08eebd24a0e1 100644
@@ -3258,7 +3258,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 				       const struct kvm_memory_slot *slot,
-				       gfn_t gfn, int max_level, bool is_private)
+				       gfn_t gfn, int max_level, bool is_gmem)
 {
 	struct kvm_lpage_info *linfo;
 	int host_level;

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -3270,7 +3270,7 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 			break;
 	}
 
-	if (is_private)
+	if (is_gmem)
 		return max_level;

I think this renaming isn't quite accurate.

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

Renaming is_private to is_gmem in this function implies that as long as gmem is used, especially for shared pages from gmem, lpage_info will always be updated and there's no need to query userspace page tables.

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3286

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +1 -2
 	if (max_level == PG_LEVEL_4K)
@@ -3283,10 +3283,9 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
 			      const struct kvm_memory_slot *slot, gfn_t gfn)
 {
-	bool is_private = kvm_slot_has_gmem(slot) &&
-			  kvm_mem_is_private(kvm, gfn);
+	bool is_gmem = kvm_slot_has_gmem(slot) && kvm_mem_from_gmem(kvm, gfn);

This renaming should probably be undone too.

Quoted 49 lines
arch/x86/kvm/mmu/mmu.c +6 -6
-	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_private);
+	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_gmem);
 }
 
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -4465,7 +4464,7 @@ static inline u8 kvm_max_level_for_order(int order)
 	return PG_LEVEL_4K;
 }
 
-static u8 kvm_max_private_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
+static u8 kvm_max_gmem_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
 					u8 max_level, int gmem_order)
 {
 	u8 req_max_level;
@@ -4491,7 +4490,7 @@ static void kvm_mmu_finish_page_fault(struct kvm_vcpu *vcpu,
 				 r == RET_PF_RETRY, fault->map_writable);
 }
 
-static int kvm_mmu_faultin_pfn_private(struct kvm_vcpu *vcpu,
+static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 				       struct kvm_page_fault *fault)
 {
 	int max_order, r;
@@ -4509,8 +4508,8 @@ static int kvm_mmu_faultin_pfn_private(struct kvm_vcpu *vcpu,
 	}
 
 	fault->map_writable = !(fault->slot->flags & KVM_MEM_READONLY);
-	fault->max_level = kvm_max_private_mapping_level(vcpu->kvm, fault->pfn,
-							 fault->max_level, max_order);
+	fault->max_level = kvm_max_gmem_mapping_level(vcpu->kvm, fault->pfn,
+						      fault->max_level, max_order);
 
 	return RET_PF_CONTINUE;
 }
@@ -4521,7 +4520,7 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	unsigned int foll = fault->write ? FOLL_WRITE : 0;
 
 	if (fault->is_private)
-		return kvm_mmu_faultin_pfn_private(vcpu, fault);
+		return kvm_mmu_faultin_pfn_gmem(vcpu, fault);
 
 	foll |= FOLL_NOWAIT;
 	fault->pfn = __kvm_faultin_pfn(fault->slot, fault->gfn, foll,
include/linux/kvm_host.h +0 -0
index d9616ee6acc7..cdcd7ac091b5 100644

include/linux/kvm_host.h static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2521

Quoted 8 lines
include/linux/kvm_host.h +5 -0
@@ -2514,6 +2514,12 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 }
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
+static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
+{
+	/* For now, only private memory gets consumed from guest_memfd. */
+	return kvm_mem_is_private(kvm, gfn);
+}

Can I understand this function as "should fault from gmem"? And hence also "was faulted from gmem"?

After this entire patch series, for arm64, KVM will always service stage 2 faults from gmem.

Perhaps this function should retain your suggested name of kvm_mem_from_gmem() but only depend on kvm_arch_gmem_supports_shared_mem(), since this patch series doesn't update the MMU in X86. So something like this,

+static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn) +{
+ return kvm_arch_gmem_supports_shared_mem(kvm);
+}

with the only usage in arm64.

When the MMU code for X86 is updated, we could then update the above with

static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn) {
- return kvm_arch_gmem_supports_shared_mem(kvm);
+ return kvm_arch_gmem_supports_shared_mem(kvm) ||
+ kvm_gmem_should_always_use_gmem(gfn_to_memslot(kvm, gfn)->gmem.file) ||
+ kvm_mem_is_private(kvm, gfn);
}

where kvm_gmem_should_always_use_gmem() will read a guest_memfd flag?

Quoted 4 lines
include/linux/kvm_host.h +1 -0
+
 #ifdef CONFIG_KVM_GMEM
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
author
#15 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Hi Ackerley,

Quoted 44 lines · Ackerley Tng
Quoted 32 lines · Fuad Tabba

Until now, faults to private memory backed by guest_memfd are always consumed from guest_memfd whereas faults to shared memory are consumed from anonymous memory. Subsequent patches will allow sharing guest_memfd backed memory in-place, and mapping it by the host. Faults to in-place shared memory should be consumed from guest_memfd as well.

In order to facilitate that, generalize the fault lookups. Currently, only private memory is consumed from guest_memfd and therefore as it stands, this patch does not change the behavior.

---
 arch/x86/kvm/mmu/mmu.c   | 19 +++++++++----------
 include/linux/kvm_host.h |  6 ++++++
 2 files changed, 15 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index 6d5dd869c890..08eebd24a0e1 100644
@@ -3258,7 +3258,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
 static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
                                     const struct kvm_memory_slot *slot,
-                                    gfn_t gfn, int max_level, bool is_private)
+                                    gfn_t gfn, int max_level, bool is_gmem)
 {
      struct kvm_lpage_info *linfo;
      int host_level;

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +1 -1
@@ -3270,7 +3270,7 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
                      break;
      }
 
-     if (is_private)
+     if (is_gmem)
              return max_level;

I think this renaming isn't quite accurate.

Quoted 9 lines

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

Renaming is_private to is_gmem in this function implies that as long as gmem is used, especially for shared pages from gmem, lpage_info will always be updated and there's no need to query userspace page tables.

I understand.

Quoted 11 lines

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3286

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +1 -2
      if (max_level == PG_LEVEL_4K)
@@ -3283,10 +3283,9 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
 int kvm_mmu_max_mapping_level(struct kvm *kvm,
                            const struct kvm_memory_slot *slot, gfn_t gfn)
 {
-     bool is_private = kvm_slot_has_gmem(slot) &&
-                       kvm_mem_is_private(kvm, gfn);
+     bool is_gmem = kvm_slot_has_gmem(slot) && kvm_mem_from_gmem(kvm, gfn);

This renaming should probably be undone too.

Ack.

Quoted 61 lines
Quoted 49 lines
arch/x86/kvm/mmu/mmu.c +6 -6
-     return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_private);
+     return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_gmem);
 }
 
 void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
@@ -4465,7 +4464,7 @@ static inline u8 kvm_max_level_for_order(int order)
      return PG_LEVEL_4K;
 }
 
-static u8 kvm_max_private_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
+static u8 kvm_max_gmem_mapping_level(struct kvm *kvm, kvm_pfn_t pfn,
                                      u8 max_level, int gmem_order)
 {
      u8 req_max_level;
@@ -4491,7 +4490,7 @@ static void kvm_mmu_finish_page_fault(struct kvm_vcpu *vcpu,
                               r == RET_PF_RETRY, fault->map_writable);
 }
 
-static int kvm_mmu_faultin_pfn_private(struct kvm_vcpu *vcpu,
+static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
                                     struct kvm_page_fault *fault)
 {
      int max_order, r;
@@ -4509,8 +4508,8 @@ static int kvm_mmu_faultin_pfn_private(struct kvm_vcpu *vcpu,
      }
 
      fault->map_writable = !(fault->slot->flags & KVM_MEM_READONLY);
-     fault->max_level = kvm_max_private_mapping_level(vcpu->kvm, fault->pfn,
-                                                      fault->max_level, max_order);
+     fault->max_level = kvm_max_gmem_mapping_level(vcpu->kvm, fault->pfn,
+                                                   fault->max_level, max_order);
 
      return RET_PF_CONTINUE;
 }
@@ -4521,7 +4520,7 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
      unsigned int foll = fault->write ? FOLL_WRITE : 0;
 
      if (fault->is_private)
-             return kvm_mmu_faultin_pfn_private(vcpu, fault);
+             return kvm_mmu_faultin_pfn_gmem(vcpu, fault);
 
      foll |= FOLL_NOWAIT;
      fault->pfn = __kvm_faultin_pfn(fault->slot, fault->gfn, foll,
include/linux/kvm_host.h +0 -0
index d9616ee6acc7..cdcd7ac091b5 100644

include/linux/kvm_host.h static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2521

Quoted 8 lines
include/linux/kvm_host.h +5 -0
@@ -2514,6 +2514,12 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 }
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
+static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
+{
+     /* For now, only private memory gets consumed from guest_memfd. */
+     return kvm_mem_is_private(kvm, gfn);
+}

Can I understand this function as "should fault from gmem"? And hence also "was faulted from gmem"?

Quoted 7 lines

After this entire patch series, for arm64, KVM will always service stage 2 faults from gmem.

Perhaps this function should retain your suggested name of kvm_mem_from_gmem() but only depend on kvm_arch_gmem_supports_shared_mem(), since this patch series doesn't update the MMU in X86. So something like this,

Ack.

Quoted 18 lines
include/linux/kvm_host.h +4 -0
+static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
+{
+       return kvm_arch_gmem_supports_shared_mem(kvm);
+}

with the only usage in arm64.

When the MMU code for X86 is updated, we could then update the above with

static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn) {

(quoted diff) +3 -1
-       return kvm_arch_gmem_supports_shared_mem(kvm);
+       return kvm_arch_gmem_supports_shared_mem(kvm) ||
+              kvm_gmem_should_always_use_gmem(gfn_to_memslot(kvm, gfn)->gmem.file) ||
+              kvm_mem_is_private(kvm, gfn);

}

Quoted 1 line

where kvm_gmem_should_always_use_gmem() will read a guest_memfd flag?

I'm not sure I follow this one... Could you please explain what you mean a bit more?

Thanks,
/fuad

Quoted 4 lines
Quoted 4 lines
include/linux/kvm_host.h +1 -0
+
 #ifdef CONFIG_KVM_GMEM
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
                   gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
#16 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

In reply to Ackerley Tng
Quoted 37 lines · Ackerley Tng
Quoted 35 lines · Fuad Tabba

Until now, faults to private memory backed by guest_memfd are always consumed from guest_memfd whereas faults to shared memory are consumed from anonymous memory. Subsequent patches will allow sharing guest_memfd backed memory in-place, and mapping it by the host. Faults to in-place shared memory should be consumed from guest_memfd as well.

In order to facilitate that, generalize the fault lookups. Currently, only private memory is consumed from guest_memfd and therefore as it stands, this patch does not change the behavior.

---
  arch/x86/kvm/mmu/mmu.c   | 19 +++++++++----------
  include/linux/kvm_host.h |  6 ++++++
  2 files changed, 15 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index 6d5dd869c890..08eebd24a0e1 100644
@@ -3258,7 +3258,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn,
 
  static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
  				       const struct kvm_memory_slot *slot,
-				       gfn_t gfn, int max_level, bool is_private)
+				       gfn_t gfn, int max_level, bool is_gmem)
  {
  	struct kvm_lpage_info *linfo;
  	int host_level;
@@ -3270,7 +3270,7 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
  			break;
  	}
 
Quoted 5 lines

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +1 -1
-	if (is_private)
+	if (is_gmem)
  		return max_level;

I think this renaming isn't quite accurate.

After our discussion yesterday, does that still hold true?

Quoted 5 lines

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

I think the reason was that: for private we won't be walking the user space pages tables.

Once guest_memfd is also responsible for the shared part, why should this here still be private-only, and why should we consider querying a user space mapping that might not even exist?

Quoted 4 lines

Renaming is_private to is_gmem in this function implies that as long as gmem is used, especially for shared pages from gmem, lpage_info will always be updated and there's no need to query userspace page tables.

I'm missing the point why shared memory from gmem should be treated differently here. Can you maybe clarify which issue you see?

Quoted 17 lines

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3286

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +1 -2
  	if (max_level == PG_LEVEL_4K)
@@ -3283,10 +3283,9 @@ static int __kvm_mmu_max_mapping_level(struct kvm *kvm,
  int kvm_mmu_max_mapping_level(struct kvm *kvm,
  			      const struct kvm_memory_slot *slot, gfn_t gfn)
  {
-	bool is_private = kvm_slot_has_gmem(slot) &&
-			  kvm_mem_is_private(kvm, gfn);
+	bool is_gmem = kvm_slot_has_gmem(slot) && kvm_mem_from_gmem(kvm, gfn);

This renaming should probably be undone too.

Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +1 -1
-	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_private);
+	return __kvm_mmu_max_mapping_level(kvm, slot, gfn, PG_LEVEL_NUM, is_gmem);
  }
Signature · 3 lines
Cheers,

David / dhildenb
#17 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 8 lines · David Hildenbrand
Quoted 5 lines · Ackerley Tng

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +1 -1
-	if (is_private)
+	if (is_gmem)
  		return max_level;

I think this renaming isn't quite accurate.

After our discussion yesterday, does that still hold true?

No.

Quoted 11 lines
Quoted 4 lines

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

I think the reason was that: for private we won't be walking the user space pages tables.

Once guest_memfd is also responsible for the shared part, why should this here still be private-only, and why should we consider querying a user space mapping that might not even exist?

+1, one of the big selling points for guest_memfd beyond CoCo is that it provides guest-first memory. It is very explicitly an intended feature that the guest mappings KVM creates can be a superset of the host userspace mappings. E.g. the guest can use larger page sizes, have RW while the host has RO, etc.

#18 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 20 lines · Sean Christopherson
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · Ackerley Tng

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +1 -1
-	if (is_private)
+	if (is_gmem)
  		return max_level;

I think this renaming isn't quite accurate.

After our discussion yesterday, does that still hold true?

No.

Quoted 8 lines
Quoted 4 lines

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

I think the reason was that: for private we won't be walking the user space pages tables.

Quoted 8 lines
Quoted 3 lines

Once guest_memfd is also responsible for the shared part, why should this here still be private-only, and why should we consider querying a user space mapping that might not even exist?

+1, one of the big selling points for guest_memfd beyond CoCo is that it provides guest-first memory. It is very explicitly an intended feature that the guest mappings KVM creates can be a superset of the host userspace mappings. E.g. the guest can use larger page sizes, have RW while the host has RO, etc.

Do you mean that __kvm_mmu_max_mapping_level() should, in addition to the parameter renaming from is_private to is_gmem, do something like

if (is_gmem)

	return kvm_gmem_get_max_mapping_level(slot, gfn);

and basically defer to gmem as long as gmem should be used for this gfn?

There is another call to __kvm_mmu_max_mapping_level() via kvm_mmu_max_mapping_level() beginning from recover_huge_pages_range(), and IIUC that doesn't go through guest_memfd.

Hence, unlike the call to __kvm_mmu_max_mapping_level() from the KVM x86 MMU fault path, guest_memfd didn't get a chance to provide its input in the form of returning max_order from kvm_gmem_get_pfn().

#19 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 31 lines · Ackerley Tng
Quoted 20 lines · Sean Christopherson
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · Ackerley Tng

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +1 -1
-	if (is_private)
+	if (is_gmem)
   		return max_level;

I think this renaming isn't quite accurate.

After our discussion yesterday, does that still hold true?

No.

Quoted 8 lines
Quoted 4 lines

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

I think the reason was that: for private we won't be walking the user space pages tables.

Quoted 8 lines
Quoted 3 lines

Once guest_memfd is also responsible for the shared part, why should this here still be private-only, and why should we consider querying a user space mapping that might not even exist?

+1, one of the big selling points for guest_memfd beyond CoCo is that it provides guest-first memory. It is very explicitly an intended feature that the guest mappings KVM creates can be a superset of the host userspace mappings. E.g. the guest can use larger page sizes, have RW while the host has RO, etc.

Quoted 5 lines

Do you mean that __kvm_mmu_max_mapping_level() should, in addition to the parameter renaming from is_private to is_gmem, do something like

if (is_gmem)

	return kvm_gmem_get_max_mapping_level(slot, gfn);

I assume you mean, not looking at lpage_info at all?

I have limited understanding what lpage_info is or what it does. I believe all it adds is a mechanism to *disable* large page mappings.

We want to disable large pages if (using 2M region as example)

(a) Mixed memory attributes. If a PFN falls into a 2M region, and parts

    of that region are shared vs. private (mixed memory attributes ->
    KVM_LPAGE_MIXED_FLAG)

-> With gmem-shared we could have mixed memory attributes, not a PFN
fracturing. (PFNs don't depend on memory attributes)

(b) page track: intercepting (mostly write) access to GFNs

So, I wonder if we still have to take care of lpage_info, at least for
handling (b) correctly [I assume so]. Regarding (a) I am not sure: once memory attributes are handled by gmem in the gmem-shared case. IIRC, with AMD SEV we might still have to honor it? But gmem itself could handle that.

What we could definitely do here for now is:

if (is_gmem)

	/* gmem only supports 4k pages for now. */
	return PG_LEVEL_4K;

And not worry about lpage_infor for the time being, until we actually do support larger pages.

Quoted 10 lines

and basically defer to gmem as long as gmem should be used for this gfn?

There is another call to __kvm_mmu_max_mapping_level() via kvm_mmu_max_mapping_level() beginning from recover_huge_pages_range(), and IIUC that doesn't go through guest_memfd.

Hence, unlike the call to __kvm_mmu_max_mapping_level() from the KVM x86 MMU fault path, guest_memfd didn't get a chance to provide its input in the form of returning max_order from kvm_gmem_get_pfn().

Right, we essentially say that "this is a private fault", likely assuming that we already verified earlier that the memory is also private.

[I can see that happening when the function is called through direct_page_fault()]

We could simply call kvm_mmu_max_mapping_level() from kvm_mmu_hugepage_adjust() I guess. (could possibly be optimized later)

Signature · 3 lines
Cheers,

David / dhildenb
#20 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ackerley Tng
Quoted 28 lines · Sean Christopherson
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · Ackerley Tng

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +1 -1
-	if (is_private)
+	if (is_gmem)
   		return max_level;

I think this renaming isn't quite accurate.

After our discussion yesterday, does that still hold true?

No.

Quoted 11 lines
Quoted 4 lines

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

I think the reason was that: for private we won't be walking the user space pages tables.

Once guest_memfd is also responsible for the shared part, why should this here still be private-only, and why should we consider querying a user space mapping that might not even exist?

+1, one of the big selling points for guest_memfd beyond CoCo is that it provides guest-first memory. It is very explicitly an intended feature that the guest mappings KVM creates can be a superset of the host userspace mappings. E.g. the guest can use larger page sizes, have RW while the host has RO, etc.

Quoted 5 lines
Quoted 5 lines

Do you mean that __kvm_mmu_max_mapping_level() should, in addition to the parameter renaming from is_private to is_gmem, do something like

if (is_gmem)

	return kvm_gmem_get_max_mapping_level(slot, gfn);

No, kvm_gmem_get_pfn() already provides the maximum allowed order, we "just" need to update that to constrain the max order based on shared vs. private. E.g. from the original guest_memfd hugepage support[*] (which never landed), to take care of the pgoff not being properly aligned to the memslot.

+ /*
+ * The folio can be mapped with a hugepage if and only if the folio is
+ * fully contained by the range the memslot is bound to. Note, the
+ * caller is responsible for handling gfn alignment, this only deals
+ * with the file binding.
+ */
+ huge_index = ALIGN(index, 1ull << *max_order);
+ if (huge_index < ALIGN(slot->gmem.pgoff, 1ull << *max_order) ||
+ huge_index + (1ull << *max_order) > slot->gmem.pgoff + slot->npages)
*max_order = 0;

[*] https://lore.kernel.org/all/[email protected]

Quoted 4 lines

I assume you mean, not looking at lpage_info at all?

I have limited understanding what lpage_info is or what it does. I believe all it adds is a mechanism to *disable* large page mappings.

Correct. It's a bit of a catch-all that's used by a variety of KVM x86 features to disable hugepages.

Quoted 10 lines

We want to disable large pages if (using 2M region as example)

(a) Mixed memory attributes. If a PFN falls into a 2M region, and parts

    of that region are shared vs. private (mixed memory attributes ->
    KVM_LPAGE_MIXED_FLAG)

-> With gmem-shared we could have mixed memory attributes, not a PFN
fracturing. (PFNs don't depend on memory attributes)

(b) page track: intercepting (mostly write) access to GFNs

It's also used to handle misaligned memslots (or sizes), e.g. if a 1GiB memory region spanse 1GiB+4KiB => 2GiB+4KiB, KVM will disallow 1GiB hugepages, and 2MiB hugepages for the head and tails. Or if the host virtual address isn't aligned with the guest physical address (see above for guest_memfd's role when there is no hva).

Quoted 2 lines

So, I wonder if we still have to take care of lpage_info, at least for handling (b) correctly [I assume so].

Ya, we do.

Quoted 4 lines

Regarding (a) I am not sure: once memory attributes are handled by gmem in the gmem-shared case. IIRC, with AMD SEV we might still have to honor it? But gmem itself could handle that.

Quoted 8 lines

What we could definitely do here for now is:

if (is_gmem)

	/* gmem only supports 4k pages for now. */
	return PG_LEVEL_4K;

And not worry about lpage_infor for the time being, until we actually do support larger pages.

I don't want to completely punt on this, because if it gets messy, then I want to know now and have a solution in hand, not find out N months from now.

That said, I don't expect it to be difficult. What we could punt on is performance of the lookups, which is the real reason KVM maintains the rather expensive disallow_lpage array.

And that said, memslots can only bind to one guest_memfd instance, so I don't immediately see any reason why the guest_memfd ioctl() couldn't process the slots that are bound to it. I.e. why not update KVM_LPAGE_MIXED_FLAG from the guest_memfd ioctl() instead of from KVM_SET_MEMORY_ATTRIBUTES?

#21 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 7 lines · Sean Christopherson
Quoted 3 lines

...
And not worry about lpage_infor for the time being, until we actually do support larger pages.

I don't want to completely punt on this, because if it gets messy, then I want to know now and have a solution in hand, not find out N months from now.

Quoted 8 lines

That said, I don't expect it to be difficult. What we could punt on is performance of the lookups, which is the real reason KVM maintains the rather expensive disallow_lpage array.

And that said, memslots can only bind to one guest_memfd instance, so I don't immediately see any reason why the guest_memfd ioctl() couldn't process the slots that are bound to it. I.e. why not update KVM_LPAGE_MIXED_FLAG from the guest_memfd ioctl() instead of from KVM_SET_MEMORY_ATTRIBUTES?

I am missing the point here to update KVM_LPAGE_MIXED_FLAG for the scenarios where in-place memory conversion will be supported with guest_memfd. As guest_memfd support for hugepages comes with the design that hugepages can't have mixed attributes. i.e. max_order returned by get_pfn will always have the same attributes for the folio range.

Is your suggestion around using guest_memfd ioctl() to also toggle memory attributes for the scenarios where guest_memfd instance doesn't have in-place memory conversion feature enabled?

#22 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 25 lines · Vishal Annapurve
Quoted 7 lines · Sean Christopherson
Quoted 3 lines

...
And not worry about lpage_infor for the time being, until we actually do support larger pages.

I don't want to completely punt on this, because if it gets messy, then I want to know now and have a solution in hand, not find out N months from now.

Quoted 8 lines

That said, I don't expect it to be difficult. What we could punt on is performance of the lookups, which is the real reason KVM maintains the rather expensive disallow_lpage array.

And that said, memslots can only bind to one guest_memfd instance, so I don't immediately see any reason why the guest_memfd ioctl() couldn't process the slots that are bound to it. I.e. why not update KVM_LPAGE_MIXED_FLAG from the guest_memfd ioctl() instead of from KVM_SET_MEMORY_ATTRIBUTES?

I am missing the point here to update KVM_LPAGE_MIXED_FLAG for the scenarios where in-place memory conversion will be supported with guest_memfd. As guest_memfd support for hugepages comes with the design that hugepages can't have mixed attributes. i.e. max_order returned by get_pfn will always have the same attributes for the folio range.

Quoted 3 lines

Is your suggestion around using guest_memfd ioctl() to also toggle memory attributes for the scenarios where guest_memfd instance doesn't have in-place memory conversion feature enabled?

Reading more into your response, I guess your suggestion is about covering different usecases present today and new usecases which may land in future, that rely on kvm_lpage_info for faster lookup. If so, then it should be easy to modify guest_memfd ioctl to update kvm_lpage_info as you suggested.

#23 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 19 lines · Vishal Annapurve
Quoted 18 lines · Vishal Annapurve
Quoted 15 lines · Sean Christopherson
Quoted 3 lines

...
And not worry about lpage_infor for the time being, until we actually do support larger pages.

I don't want to completely punt on this, because if it gets messy, then I want to know now and have a solution in hand, not find out N months from now.

That said, I don't expect it to be difficult. What we could punt on is performance of the lookups, which is the real reason KVM maintains the rather expensive disallow_lpage array.

And that said, memslots can only bind to one guest_memfd instance, so I don't immediately see any reason why the guest_memfd ioctl() couldn't process the slots that are bound to it. I.e. why not update KVM_LPAGE_MIXED_FLAG from the guest_memfd ioctl() instead of from KVM_SET_MEMORY_ATTRIBUTES?

Quoted 6 lines
Quoted 6 lines

I am missing the point here to update KVM_LPAGE_MIXED_FLAG for the scenarios where in-place memory conversion will be supported with guest_memfd. As guest_memfd support for hugepages comes with the design that hugepages can't have mixed attributes. i.e. max_order returned by get_pfn will always have the same attributes for the folio range.

Oh, if this will naturally be handled by guest_memfd, then do that. I was purely reacting to David's suggestion to "not worry about lpage_infor for the time being, until we actually do support larger pages".

Quoted 9 lines
Quoted 3 lines

Is your suggestion around using guest_memfd ioctl() to also toggle memory attributes for the scenarios where guest_memfd instance doesn't have in-place memory conversion feature enabled?

Reading more into your response, I guess your suggestion is about covering different usecases present today and new usecases which may land in future, that rely on kvm_lpage_info for faster lookup. If so, then it should be easy to modify guest_memfd ioctl to update kvm_lpage_info as you suggested.

Nah, I just missed/forgot that using a single guest_memfd for private and shared would naturally need to split the folio and thus this would Just Work.

#24 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 35 lines · Sean Christopherson
Quoted 19 lines · Vishal Annapurve
Quoted 18 lines · Vishal Annapurve
Quoted 15 lines · Sean Christopherson
Quoted 3 lines

...
And not worry about lpage_infor for the time being, until we actually do support larger pages.

I don't want to completely punt on this, because if it gets messy, then I want to know now and have a solution in hand, not find out N months from now.

That said, I don't expect it to be difficult. What we could punt on is performance of the lookups, which is the real reason KVM maintains the rather expensive disallow_lpage array.

And that said, memslots can only bind to one guest_memfd instance, so I don't immediately see any reason why the guest_memfd ioctl() couldn't process the slots that are bound to it. I.e. why not update KVM_LPAGE_MIXED_FLAG from the guest_memfd ioctl() instead of from KVM_SET_MEMORY_ATTRIBUTES?

Quoted 6 lines
Quoted 6 lines

I am missing the point here to update KVM_LPAGE_MIXED_FLAG for the scenarios where in-place memory conversion will be supported with guest_memfd. As guest_memfd support for hugepages comes with the design that hugepages can't have mixed attributes. i.e. max_order returned by get_pfn will always have the same attributes for the folio range.

Oh, if this will naturally be handled by guest_memfd, then do that. I was purely reacting to David's suggestion to "not worry about lpage_infor for the time being, until we actually do support larger pages".

Quoted 4 lines
Quoted 3 lines

Is your suggestion around using guest_memfd ioctl() to also toggle memory attributes for the scenarios where guest_memfd instance doesn't have in-place memory conversion feature enabled?

Quoted 8 lines
Quoted 5 lines

Reading more into your response, I guess your suggestion is about covering different usecases present today and new usecases which may land in future, that rely on kvm_lpage_info for faster lookup. If so, then it should be easy to modify guest_memfd ioctl to update kvm_lpage_info as you suggested.

Nah, I just missed/forgot that using a single guest_memfd for private and shared would naturally need to split the folio and thus this would Just Work.

Yeah, I ignored that fact as well. So essentially, this patch should be mostly good for now.

Only kvm_mmu_hugepage_adjust() must be taught to not rely on fault->is_private.

Once we support large folios in guest_memfd, only the "alignment" consideration might have to be taken into account.

Anything else?

Signature · 3 lines
Cheers,

David / dhildenb
#25 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 36 lines · David Hildenbrand
Quoted 35 lines · Sean Christopherson
Quoted 19 lines · Vishal Annapurve
Quoted 18 lines · Vishal Annapurve
Quoted 15 lines · Sean Christopherson
Quoted 3 lines

...
And not worry about lpage_infor for the time being, until we actually do support larger pages.

I don't want to completely punt on this, because if it gets messy, then I want to know now and have a solution in hand, not find out N months from now.

That said, I don't expect it to be difficult. What we could punt on is performance of the lookups, which is the real reason KVM maintains the rather expensive disallow_lpage array.

And that said, memslots can only bind to one guest_memfd instance, so I don't immediately see any reason why the guest_memfd ioctl() couldn't process the slots that are bound to it. I.e. why not update KVM_LPAGE_MIXED_FLAG from the guest_memfd ioctl() instead of from KVM_SET_MEMORY_ATTRIBUTES?

Quoted 6 lines
Quoted 6 lines

I am missing the point here to update KVM_LPAGE_MIXED_FLAG for the scenarios where in-place memory conversion will be supported with guest_memfd. As guest_memfd support for hugepages comes with the design that hugepages can't have mixed attributes. i.e. max_order returned by get_pfn will always have the same attributes for the folio range.

Oh, if this will naturally be handled by guest_memfd, then do that. I was purely reacting to David's suggestion to "not worry about lpage_infor for the time being, until we actually do support larger pages".

Quoted 4 lines
Quoted 3 lines

Is your suggestion around using guest_memfd ioctl() to also toggle memory attributes for the scenarios where guest_memfd instance doesn't have in-place memory conversion feature enabled?

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

Reading more into your response, I guess your suggestion is about covering different usecases present today and new usecases which may land in future, that rely on kvm_lpage_info for faster lookup. If so, then it should be easy to modify guest_memfd ioctl to update kvm_lpage_info as you suggested.

Nah, I just missed/forgot that using a single guest_memfd for private and shared would naturally need to split the folio and thus this would Just Work.

Sean, David, I'm circling back to make sure I'm following the discussion correctly before Fuad sends out the next revision of this series.

Quoted 4 lines

Yeah, I ignored that fact as well. So essentially, this patch should be mostly good for now.

From here [1], these changes will make it to v9

+ kvm_max_private_mapping_level renaming to kvm_max_gmem_mapping_level
+ kvm_mmu_faultin_pfn_private renaming to kvm_mmu_faultin_pfn_gmem

Quoted 3 lines

Only kvm_mmu_hugepage_adjust() must be taught to not rely on fault->is_private.

I think fault->is_private should contribute to determining the max mapping level.

By the time kvm_mmu_hugepage_adjust() is called,

* For Coco VMs using guest_memfd only for private memory,
* fault->is_private would have been checked to align with

    kvm->mem_attr_array, so 

* For Coco VMs using guest_memfd for both private/shared memory,
* fault->is_private would have been checked to align with guest_memfd's shareability
* For non-Coco VMs using guest_memfd
* fault->is_private would be false

Hence fault->is_private can be relied on when calling kvm_mmu_hugepage_adjust().

If fault->is_private, there will be no host userspace mapping to check, hence in __kvm_mmu_max_mapping_level(), we should skip querying host page tables.

If !fault->is_private, for shared memory ranges, if the VM uses guest_memfd only for shared memory, we should query host page tables.

If !fault->is_private, for shared memory ranges, if the VM uses guest_memfd for both shared/private memory, we should not query host page tables.

If !fault->is_private, for non-Coco VMs, we should not query host page tables.

I propose to rename the parameter is_private to skip_host_page_tables, so

- if (is_private)
+ if (skip_host_page_tables)

	return max_level;

and pass

skip_host_page_tables = fault->is_private ||

			kvm_gmem_memslot_supports_shared(fault->slot);

where kvm_gmem_memslot_supports_shared() checks the inode in the memslot for GUEST_MEMFD_FLAG_SUPPORT_SHARED.

For recover_huge_pages_range(), the other user of __kvm_mmu_max_mapping_level(), currently there's no prior call to kvm_gmem_get_pfn() to get max_order or max_level, so I propose to call __kvm_mmu_max_mapping_level() with

if (kvm_gmem_memslot_supports_shared(slot)) {
	max_level = kvm_gmem_max_mapping_level(slot, gfn);
	skip_host_page_tables = true;
} else {
	max_level = PG_LEVEL_NUM;
        skip_host_page_tables = kvm_slot_has_gmem(slot) &&
				kvm_mem_is_private(kvm, gfn);
}

Without 1G support, kvm_gmem_max_mapping_level(slot, gfn) would always return 4K.

With 1G support, kvm_gmem_max_mapping_level(slot, gfn) would return the level for the page's order, at the offset corresponding to the gfn.

Quoted 3 lines

Once we support large folios in guest_memfd, only the "alignment" consideration might have to be taken into account.

I'll be handling this alignment as part of the 1G page support series (won't be part of Fuad's first stage series) [2]

Quoted 6 lines

Anything else?

Signature · 3 lines
Cheers,

David / dhildenb

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

#26 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 2 lines · Ackerley Tng

Sean, David, I'm circling back to make sure I'm following the discussion correctly before Fuad sends out the next revision of this series.

Honestly, just send the next version. Try to review a description of code is an exercise in frustration. More versions of a series isn't inherently bad.

#27 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

In reply to Ackerley Tng
Quoted 19 lines
Quoted 4 lines

Yeah, I ignored that fact as well. So essentially, this patch should be mostly good for now.

From here [1], these changes will make it to v9

+ kvm_max_private_mapping_level renaming to kvm_max_gmem_mapping_level
+ kvm_mmu_faultin_pfn_private renaming to kvm_mmu_faultin_pfn_gmem

Quoted 3 lines

Only kvm_mmu_hugepage_adjust() must be taught to not rely on fault->is_private.

I think fault->is_private should contribute to determining the max mapping level.

By the time kvm_mmu_hugepage_adjust() is called,

Quoted 8 lines

* For Coco VMs using guest_memfd only for private memory,
* fault->is_private would have been checked to align with

     kvm->mem_attr_array, so

* For Coco VMs using guest_memfd for both private/shared memory,
* fault->is_private would have been checked to align with guest_memfd's shareability
* For non-Coco VMs using guest_memfd
* fault->is_private would be false

But as Sean said, looking at the code might be easier.

Maybe just send the resulting diff of the patch here real quick?

Quoted 19 lines

Hence fault->is_private can be relied on when calling kvm_mmu_hugepage_adjust().

If fault->is_private, there will be no host userspace mapping to check, hence in __kvm_mmu_max_mapping_level(), we should skip querying host page tables.

If !fault->is_private, for shared memory ranges, if the VM uses guest_memfd only for shared memory, we should query host page tables.

If !fault->is_private, for shared memory ranges, if the VM uses guest_memfd for both shared/private memory, we should not query host page tables.

Quoted 1 line

If !fault->is_private, for non-Coco VMs, we should not query host page

tables.

Quoted 1 line

I propose to rename the parameter is_private to skip_host_page_tables,

so

Quoted 9 lines

- if (is_private)
+ if (skip_host_page_tables)

	return max_level;

and pass

skip_host_page_tables = fault->is_private ||

			kvm_gmem_memslot_supports_shared(fault->slot);

How is that better than calling it "is_gmem" / "from_gmem" etc? :) Anyhow, no strong opinion, spelling out that something is from gmem implies that we don't care about page tables.

Quoted 2 lines

where kvm_gmem_memslot_supports_shared() checks the inode in the memslot for GUEST_MEMFD_FLAG_SUPPORT_SHARED.

Makes sense.

Signature · 3 lines
Cheers,

David / dhildenb
#28 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

In reply to Ackerley Tng
Quoted 14 lines · Ackerley Tng

From here [1], these changes will make it to v9

+ kvm_max_private_mapping_level renaming to kvm_max_gmem_mapping_level
+ kvm_mmu_faultin_pfn_private renaming to kvm_mmu_faultin_pfn_gmem

Quoted 3 lines

Only kvm_mmu_hugepage_adjust() must be taught to not rely on fault->is_private.

I think fault->is_private should contribute to determining the max mapping level.

By the time kvm_mmu_hugepage_adjust() is called,

Quoted 8 lines

* For Coco VMs using guest_memfd only for private memory,
* fault->is_private would have been checked to align with

    kvm->mem_attr_array, so

* For Coco VMs using guest_memfd for both private/shared memory,
* fault->is_private would have been checked to align with guest_memfd's shareability
* For non-Coco VMs using guest_memfd
* fault->is_private would be false

I'm not sure exactly which thread to respond to, but it seems like the idea now is to have a *VM* flag determine if shared faults use gmem or use the user mappings. It seems more natural for that to be a property of the memslot / a *memslot* flag.

Sean, Fuad, what do you think? I don't see any downsides, and it seems strictly more flexible.

Quoted 2 lines

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

#29 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 24 lines · James Houghton
Quoted 14 lines · Ackerley Tng

From here [1], these changes will make it to v9

+ kvm_max_private_mapping_level renaming to kvm_max_gmem_mapping_level
+ kvm_mmu_faultin_pfn_private renaming to kvm_mmu_faultin_pfn_gmem

Quoted 3 lines

Only kvm_mmu_hugepage_adjust() must be taught to not rely on fault->is_private.

I think fault->is_private should contribute to determining the max mapping level.

By the time kvm_mmu_hugepage_adjust() is called,

Quoted 8 lines

* For Coco VMs using guest_memfd only for private memory,
* fault->is_private would have been checked to align with

     kvm->mem_attr_array, so

* For Coco VMs using guest_memfd for both private/shared memory,
* fault->is_private would have been checked to align with guest_memfd's shareability
* For non-Coco VMs using guest_memfd
* fault->is_private would be false

Quoted 4 lines

I'm not sure exactly which thread to respond to, but it seems like the idea now is to have a *VM* flag determine if shared faults use gmem or use the user mappings. It seems more natural for that to be a property of the memslot / a *memslot* flag.

I think that's exactly what we discussed in the last meetings. The guest_memfd flag essentially defines that.

So it's not strictly a memslot flag but rather a guest_memfd flag, and the memslot is configured with that guest_memfd, inheriting that flag.

There might be a VM capability, whether it supports creation of these new guest_memfds (iow, guest_memfd understands the new flag).

Signature · 3 lines
Cheers,

David / dhildenb
#30 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 31 lines · David Hildenbrand
Quoted 24 lines · James Houghton
Quoted 14 lines · Ackerley Tng

From here [1], these changes will make it to v9

+ kvm_max_private_mapping_level renaming to kvm_max_gmem_mapping_level
+ kvm_mmu_faultin_pfn_private renaming to kvm_mmu_faultin_pfn_gmem

Quoted 3 lines

Only kvm_mmu_hugepage_adjust() must be taught to not rely on fault->is_private.

I think fault->is_private should contribute to determining the max mapping level.

By the time kvm_mmu_hugepage_adjust() is called,

Quoted 8 lines

* For Coco VMs using guest_memfd only for private memory,
* fault->is_private would have been checked to align with

     kvm->mem_attr_array, so

* For Coco VMs using guest_memfd for both private/shared memory,
* fault->is_private would have been checked to align with guest_memfd's shareability
* For non-Coco VMs using guest_memfd
* fault->is_private would be false

Quoted 4 lines

I'm not sure exactly which thread to respond to, but it seems like the idea now is to have a *VM* flag determine if shared faults use gmem or use the user mappings. It seems more natural for that to be a property of the memslot / a *memslot* flag.

Quoted 8 lines

I think that's exactly what we discussed in the last meetings. The guest_memfd flag essentially defines that.

So it's not strictly a memslot flag but rather a guest_memfd flag, and the memslot is configured with that guest_memfd, inheriting that flag.

There might be a VM capability, whether it supports creation of these new guest_memfds (iow, guest_memfd understands the new flag).

Oh yeah, I remember now, thanks for clearing that up for me. And I can see it in the notes from last week's guest_memfd meeting. :)

#31 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

In reply to Sean Christopherson
Quoted 18 lines · Sean Christopherson

<snip>

... we "just" need
to update that to constrain the max order based on shared vs. private. E.g. from the original guest_memfd hugepage support[*] (which never landed), to take care of the pgoff not being properly aligned to the memslot.

(quoted diff) +9 -0
+	/*
+	 * The folio can be mapped with a hugepage if and only if the folio is
+	 * fully contained by the range the memslot is bound to.  Note, the
+	 * caller is responsible for handling gfn alignment, this only deals
+	 * with the file binding.
+	 */
+	huge_index = ALIGN(index, 1ull << *max_order);
+	if (huge_index < ALIGN(slot->gmem.pgoff, 1ull << *max_order) ||
+	    huge_index + (1ull << *max_order) > slot->gmem.pgoff + slot->npages)
 		*max_order = 0;
Quoted 2 lines

[*] https://lore.kernel.org/all/[email protected]

Regarding this alignment check, did you also consider checking at memslot binding time? Would this [1] work/be better?

[1] https://lore.kernel.org/all/[email protected]/

#32 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

In reply to David Hildenbrand
Quoted 32 lines · David Hildenbrand
Quoted 31 lines · Ackerley Tng
Quoted 28 lines · Sean Christopherson
Quoted 8 lines · David Hildenbrand
Quoted 5 lines · Ackerley Tng

arch/x86/kvm/mmu/mmu.c static int __kvm_mmu_max_mapping_level(struct kvm *kvm, · line 3274

Quoted 3 lines
arch/x86/kvm/mmu/mmu.c +1 -1
-	if (is_private)
+	if (is_gmem)
   		return max_level;

I think this renaming isn't quite accurate.

After our discussion yesterday, does that still hold true?

No.

Quoted 11 lines
Quoted 4 lines

IIUC in __kvm_mmu_max_mapping_level(), we skip considering host_pfn_mapping_level() if the gfn is private because private memory will not be mapped to userspace, so there's no need to query userspace page tables in host_pfn_mapping_level().

I think the reason was that: for private we won't be walking the user space pages tables.

Once guest_memfd is also responsible for the shared part, why should this here still be private-only, and why should we consider querying a user space mapping that might not even exist?

+1, one of the big selling points for guest_memfd beyond CoCo is that it provides guest-first memory. It is very explicitly an intended feature that the guest mappings KVM creates can be a superset of the host userspace mappings. E.g. the guest can use larger page sizes, have RW while the host has RO, etc.

Quoted 8 lines
Quoted 5 lines

Do you mean that __kvm_mmu_max_mapping_level() should, in addition to the parameter renaming from is_private to is_gmem, do something like

if (is_gmem)

	return kvm_gmem_get_max_mapping_level(slot, gfn);

I assume you mean, not looking at lpage_info at all?

My bad. I actually meant just to take input from guest_memfd and stop there without checking with host page tables, perhaps something like min(kvm_gmem_get_max_mapping_level(slot, gfn), max_level);

Quoted 3 lines

I have limited understanding what lpage_info is or what it does. I believe all it adds is a mechanism to *disable* large page mappings.

This is my understanding too.

Quoted 11 lines

We want to disable large pages if (using 2M region as example)

(a) Mixed memory attributes. If a PFN falls into a 2M region, and parts

     of that region are shared vs. private (mixed memory attributes ->
     KVM_LPAGE_MIXED_FLAG)

-> With gmem-shared we could have mixed memory attributes, not a PFN
fracturing. (PFNs don't depend on memory attributes)

(b) page track: intercepting (mostly write) access to GFNs

Could you explain more about page track case?

Quoted 7 lines

So, I wonder if we still have to take care of lpage_info, at least for handling (b) correctly [I assume so]. Regarding (a) I am not sure: once memory attributes are handled by gmem in the gmem-shared case. IIRC, with AMD SEV we might still have to honor it? But gmem itself could handle that.

For AMD SEV, I believe kvm_max_private_mapping_level() already takes care of that, at least for the MMU faulting path [1], where guest_memfd gives input using max_order, then arch-specific callback contributes input.

Quoted 11 lines

What we could definitely do here for now is:

if (is_gmem)

	/* gmem only supports 4k pages for now. */
	return PG_LEVEL_4K;

And not worry about lpage_infor for the time being, until we actually do support larger pages.

Perhaps this is better explained as an RFC in code. I'll put in a patch as part of Fuad's series if Fuad doesn't mind.

Quoted 11 lines
Quoted 10 lines

and basically defer to gmem as long as gmem should be used for this gfn?

There is another call to __kvm_mmu_max_mapping_level() via kvm_mmu_max_mapping_level() beginning from recover_huge_pages_range(), and IIUC that doesn't go through guest_memfd.

Hence, unlike the call to __kvm_mmu_max_mapping_level() from the KVM x86 MMU fault path, guest_memfd didn't get a chance to provide its input in the form of returning max_order from kvm_gmem_get_pfn().

Quoted 13 lines

Right, we essentially say that "this is a private fault", likely assuming that we already verified earlier that the memory is also private.

[I can see that happening when the function is called through direct_page_fault()]

We could simply call kvm_mmu_max_mapping_level() from kvm_mmu_hugepage_adjust() I guess. (could possibly be optimized later)

Signature · 3 lines
Cheers,

David / dhildenb

[1] https://github.com/torvalds/linux/blob/01f95500a162fca88cefab9ed64ceded5afabc12/arch/x86/kvm/mmu/mmu.c#L4480

#33 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

Quoted 7 lines · Ackerley Tng
Quoted 7 lines · Ackerley Tng

We want to disable large pages if (using 2M region as example)

(a) Mixed memory attributes. If a PFN falls into a 2M region, and parts

     of that region are shared vs. private (mixed memory attributes ->
     KVM_LPAGE_MIXED_FLAG)
Quoted 7 lines
Quoted 5 lines

-> With gmem-shared we could have mixed memory attributes, not a PFN
fracturing. (PFNs don't depend on memory attributes)

(b) page track: intercepting (mostly write) access to GFNs

Could you explain more about page track case?

KVM disallows hugepages when shadowing a gfn, because write-protecting a 2MiB (let alone a 1GiB) page would be insanely expensive, as KVM would need to intercept and emulate an absurd number of instructions that have nothing to do with the guest's page tables.

Reviewed-by on v8 6/13
#34 lore raw

Re: [PATCH v8 06/13] KVM: x86: Generalize private fault lookups to guest_memfd fault lookups

In reply to Fuad Tabba
Quoted 6 lines · Fuad Tabba

Until now, faults to private memory backed by guest_memfd are always consumed from guest_memfd whereas faults to shared memory are consumed from anonymous memory. Subsequent patches will allow sharing guest_memfd backed memory in-place, and mapping it by the host. Faults to in-place shared memory should be consumed from guest_memfd as well.

Quoted 7 lines

In order to facilitate that, generalize the fault lookups. Currently, only private memory is consumed from guest_memfd and therefore as it stands, this patch does not change the behavior.

[snip]

author
#35 lore raw

PATCH v8 7/13 KVM: Fix comments that refer to slots_lock

In reply to Fuad Tabba

Fix comments so that they refer to slots_lock instead of slots_locks (remove trailing s).

---
 include/linux/kvm_host.h | 2 +-
 virt/kvm/kvm_main.c      | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
include/linux/kvm_host.h +1 -1
index cdcd7ac091b5..9419fb99f7c2 100644
@@ -859,7 +859,7 @@ struct kvm {
 	struct notifier_block pm_notifier;
 #endif
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
-	/* Protected by slots_locks (for writes) and RCU (for reads) */
+	/* Protected by slots_lock (for writes) and RCU (for reads) */
 	struct xarray mem_attr_array;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
virt/kvm/kvm_main.c +1 -1
index 2468d50a9ed4..6289ea1685dd 100644
@@ -333,7 +333,7 @@ void kvm_flush_remote_tlbs_memslot(struct kvm *kvm,
 	 * All current use cases for flushing the TLBs for a specific memslot
 	 * are related to dirty logging, and many do the TLB flush out of
 	 * mmu_lock. The interaction between the various operations on memslot
-	 * must be serialized by slots_locks to ensure the TLB flush from one
+	 * must be serialized by slots_lock to ensure the TLB flush from one
 	 * operation is observed by any other operation on the same memslot.
 	 */
 	lockdep_assert_held(&kvm->slots_lock);
Signature · 1 line
2.49.0.901.g37484f566f-goog
author
#39 lore raw

PATCH v8 8/13 KVM: guest_memfd: Allow host to map guest_memfd() pages

In reply to Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

---
 include/linux/kvm_host.h | 15 ++++++-
 include/uapi/linux/kvm.h |  1 +
 virt/kvm/Kconfig         |  5 +++
 virt/kvm/guest_memfd.c   | 92 ++++++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      |  4 ++
 5 files changed, 116 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +14 -1
index 9419fb99f7c2..f3af6bff3232 100644
@@ -729,6 +729,17 @@ static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
 }
 #endif
 
+/*
+ * Arch code must define kvm_arch_gmem_supports_shared_mem if support for
+ * private memory is enabled and it supports in-place shared/private conversion.
+ */
+#if !defined(kvm_arch_gmem_supports_shared_mem) && !IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM)
+static inline bool kvm_arch_gmem_supports_shared_mem(struct kvm *kvm)
+{
+	return false;
+}
+#endif
+
 #ifndef kvm_arch_has_readonly_mem
 static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm)
 {
@@ -2516,7 +2527,9 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 
 static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
 {
-	/* For now, only private memory gets consumed from guest_memfd. */
+	if (kvm_arch_gmem_supports_shared_mem(kvm))
+		return true;
+
 	return kvm_mem_is_private(kvm, gfn);
 }
 
include/uapi/linux/kvm.h +1 -0
index b6ae8ad8934b..8bc8046c7f3a 100644
@@ -930,6 +930,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237
 #define KVM_CAP_X86_GUEST_MODE 238
 #define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239
+#define KVM_CAP_GMEM_SHARED_MEM 240
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
virt/kvm/Kconfig +5 -0
index 559c93ad90be..f4e469a62a60 100644
@@ -128,3 +128,8 @@ config HAVE_KVM_ARCH_GMEM_PREPARE
 config HAVE_KVM_ARCH_GMEM_INVALIDATE
        bool
        depends on KVM_GMEM
+
+config KVM_GMEM_SHARED_MEM
+       select KVM_GMEM
+       bool
+       prompt "Enables in-place shared memory for guest_memfd"
virt/kvm/guest_memfd.c +92 -0

99 lines not on this page · raw message

virt/kvm/kvm_main.c +4 -0
index 6289ea1685dd..c75d8e188eb7 100644
@@ -4845,6 +4845,10 @@ static int kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
 #ifdef CONFIG_KVM_GMEM
 	case KVM_CAP_GUEST_MEMFD:
 		return !kvm || kvm_arch_supports_gmem(kvm);
+#endif
+#ifdef CONFIG_KVM_GMEM_SHARED_MEM
+	case KVM_CAP_GMEM_SHARED_MEM:
+		return !kvm || kvm_arch_gmem_supports_shared_mem(kvm);
 #endif
 	default:
 		break;
Signature · 1 line
2.49.0.901.g37484f566f-goog
#40 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Quoted 6 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

Quoted 7 lines

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

I think you should clarify here that it's not about "supports in-place conversion" in the context of this series.

It's about mapping shared pages only; initially, we'll introduce the option to only have shared memory in guest memfd, and later we'll introduce the option for in-place conversion.

Signature · 3 lines
Cheers,

David / dhildenb
author
#41 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Quoted 16 lines · David Hildenbrand
Quoted 6 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

Quoted 7 lines

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

Quoted 6 lines

I think you should clarify here that it's not about "supports in-place conversion" in the context of this series.

It's about mapping shared pages only; initially, we'll introduce the option to only have shared memory in guest memfd, and later we'll introduce the option for in-place conversion.

That's right. I'll fix this.

Thanks,
/fuad

Quoted 5 lines

--
Cheers,

David / dhildenb

#42 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

In reply to Fuad Tabba
Quoted 46 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

---
  include/linux/kvm_host.h | 15 ++++++-
  include/uapi/linux/kvm.h |  1 +
  virt/kvm/Kconfig         |  5 +++
  virt/kvm/guest_memfd.c   | 92 ++++++++++++++++++++++++++++++++++++++++
  virt/kvm/kvm_main.c      |  4 ++
  5 files changed, 116 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +11 -0
index 9419fb99f7c2..f3af6bff3232 100644
@@ -729,6 +729,17 @@ static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
  }
  #endif
 
+/*
+ * Arch code must define kvm_arch_gmem_supports_shared_mem if support for
+ * private memory is enabled and it supports in-place shared/private conversion.
+ */
+#if !defined(kvm_arch_gmem_supports_shared_mem) && !IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM)
+static inline bool kvm_arch_gmem_supports_shared_mem(struct kvm *kvm)
+{
+	return false;
+}
+#endif
+
  #ifndef kvm_arch_has_readonly_mem
  static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm)
  {

include/linux/kvm_host.h static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2531

Quoted 6 lines
include/linux/kvm_host.h +2 -1
@@ -2516,7 +2527,9 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 
  static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
  {
-	/* For now, only private memory gets consumed from guest_memfd. */
+	if (kvm_arch_gmem_supports_shared_mem(kvm))
+		return true;

After our discussion yesterday, am I correct that we will not be querying the KVM capability, but instead the "SHARED_TRACKING" (or however that flag is called) on the underlying guest_memfd instead?

I assume the function would then look something like

if (!kvm_supports_gmem(kvm))

	return false;

if (kvm_arch_gmem_supports_shared_mem(kvm))
return .. TBD, test the gmem flag for the slot via gfn
return kvm_mem_is_private(kvm, gfn);

Quoted 37 lines
include/linux/kvm_host.h +1 -0
+
  	return kvm_mem_is_private(kvm, gfn);
  }
 
include/uapi/linux/kvm.h +1 -0
index b6ae8ad8934b..8bc8046c7f3a 100644
@@ -930,6 +930,7 @@ struct kvm_enable_cap {
  #define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237
  #define KVM_CAP_X86_GUEST_MODE 238
  #define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239
+#define KVM_CAP_GMEM_SHARED_MEM 240
 
  struct kvm_irq_routing_irqchip {
  	__u32 irqchip;
virt/kvm/Kconfig +5 -0
index 559c93ad90be..f4e469a62a60 100644
@@ -128,3 +128,8 @@ config HAVE_KVM_ARCH_GMEM_PREPARE
  config HAVE_KVM_ARCH_GMEM_INVALIDATE
         bool
         depends on KVM_GMEM
+
+config KVM_GMEM_SHARED_MEM
+       select KVM_GMEM
+       bool
+       prompt "Enables in-place shared memory for guest_memfd"
virt/kvm/guest_memfd.c +0 -0
index 6db515833f61..8bc8fc991d58 100644
@@ -312,7 +312,99 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
  	return gfn - slot->base_gfn + slot->gmem.pgoff;
  }
 

virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 324

Quoted 10 lines
virt/kvm/guest_memfd.c +10 -0
+#ifdef CONFIG_KVM_GMEM_SHARED_MEM
+/*
+ * Returns true if the folio is shared with the host and the guest.
+ */
+static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index)
+{
+	struct kvm_gmem *gmem = file->private_data;
+
+	/* For now, VMs that support shared memory share all their memory. */
+	return kvm_arch_gmem_supports_shared_mem(gmem->kvm);

Similar here: likely we want to check the guest_memfd flag.

Signature · 3 lines
Cheers,

David / dhildenb
#43 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Quoted 58 lines · David Hildenbrand
Quoted 46 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

---
  include/linux/kvm_host.h | 15 ++++++-
  include/uapi/linux/kvm.h |  1 +
  virt/kvm/Kconfig         |  5 +++
  virt/kvm/guest_memfd.c   | 92 ++++++++++++++++++++++++++++++++++++++++
  virt/kvm/kvm_main.c      |  4 ++
  5 files changed, 116 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +11 -0
index 9419fb99f7c2..f3af6bff3232 100644
@@ -729,6 +729,17 @@ static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
  }
  #endif
 
+/*
+ * Arch code must define kvm_arch_gmem_supports_shared_mem if support for
+ * private memory is enabled and it supports in-place shared/private conversion.
+ */
+#if !defined(kvm_arch_gmem_supports_shared_mem) && !IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM)
+static inline bool kvm_arch_gmem_supports_shared_mem(struct kvm *kvm)
+{
+	return false;
+}
+#endif
+
  #ifndef kvm_arch_has_readonly_mem
  static inline bool kvm_arch_has_readonly_mem(struct kvm *kvm)
  {

include/linux/kvm_host.h static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn) · line 2531

Quoted 6 lines
include/linux/kvm_host.h +2 -1
@@ -2516,7 +2527,9 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 
  static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
  {
-	/* For now, only private memory gets consumed from guest_memfd. */
+	if (kvm_arch_gmem_supports_shared_mem(kvm))
+		return true;

After our discussion yesterday, am I correct that we will not be querying the KVM capability, but instead the "SHARED_TRACKING" (or however that flag is called) on the underlying guest_memfd instead?

Quoted 8 lines

I assume the function would then look something like

if (!kvm_supports_gmem(kvm))

	return false;

if (kvm_arch_gmem_supports_shared_mem(kvm))
return .. TBD, test the gmem flag for the slot via gfn
return kvm_mem_is_private(kvm, gfn);

Yes, I believe we're aligned here. I added a patch that will do this, but it depends on other parts of the patch series and I think it's better if you review it altogether when Fuad posts it (as opposed to reviewing a snippet I drop here.

Quoted 54 lines
Quoted 37 lines
include/linux/kvm_host.h +1 -0
+
  	return kvm_mem_is_private(kvm, gfn);
  }
 
include/uapi/linux/kvm.h +1 -0
index b6ae8ad8934b..8bc8046c7f3a 100644
@@ -930,6 +930,7 @@ struct kvm_enable_cap {
  #define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237
  #define KVM_CAP_X86_GUEST_MODE 238
  #define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239
+#define KVM_CAP_GMEM_SHARED_MEM 240
 
  struct kvm_irq_routing_irqchip {
  	__u32 irqchip;
virt/kvm/Kconfig +5 -0
index 559c93ad90be..f4e469a62a60 100644
@@ -128,3 +128,8 @@ config HAVE_KVM_ARCH_GMEM_PREPARE
  config HAVE_KVM_ARCH_GMEM_INVALIDATE
         bool
         depends on KVM_GMEM
+
+config KVM_GMEM_SHARED_MEM
+       select KVM_GMEM
+       bool
+       prompt "Enables in-place shared memory for guest_memfd"
virt/kvm/guest_memfd.c +0 -0
index 6db515833f61..8bc8fc991d58 100644
@@ -312,7 +312,99 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
  	return gfn - slot->base_gfn + slot->gmem.pgoff;
  }
 

virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 324

Quoted 10 lines
virt/kvm/guest_memfd.c +10 -0
+#ifdef CONFIG_KVM_GMEM_SHARED_MEM
+/*
+ * Returns true if the folio is shared with the host and the guest.
+ */
+static bool kvm_gmem_offset_is_shared(struct file *file, pgoff_t index)
+{
+	struct kvm_gmem *gmem = file->private_data;
+
+	/* For now, VMs that support shared memory share all their memory. */
+	return kvm_arch_gmem_supports_shared_mem(gmem->kvm);

Similar here: likely we want to check the guest_memfd flag.

Signature · 3 lines
Cheers,

David / dhildenb
#44 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

In reply to Fuad Tabba
Quoted 23 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

---
 include/linux/kvm_host.h | 15 ++++++-
 include/uapi/linux/kvm.h |  1 +
 virt/kvm/Kconfig         |  5 +++
 virt/kvm/guest_memfd.c   | 92 ++++++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      |  4 ++
 5 files changed, 116 insertions(+), 1 deletion(-)
Quoted 1 line

<snip>

At the guest_memfd call on 2025-05-01, we discussed that if guest_memfd is created with GUEST_MEMFD_FLAG_SUPPORT_SHARED set, then if slot->userspace_addr != 0, we would validate that the folio slot->userspace_addr points to matches up with the folio guest_memfd would return for the same offset.

I can think of one way to do this validation, which is to call KVM's hva_to_pfn() function and then call kvm_gmem_get_folio() on the fd and offset, and then check that the PFNs are equal.

However, that would cause the page to be allocated. Any ideas on how we could do this validation without allocating the page?

#45 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Quoted 25 lines · Ackerley Tng
Quoted 23 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

---
 include/linux/kvm_host.h | 15 ++++++-
 include/uapi/linux/kvm.h |  1 +
 virt/kvm/Kconfig         |  5 +++
 virt/kvm/guest_memfd.c   | 92 ++++++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      |  4 ++
 5 files changed, 116 insertions(+), 1 deletion(-)
Quoted 7 lines
Quoted 1 line

<snip>

At the guest_memfd call on 2025-05-01, we discussed that if guest_memfd is created with GUEST_MEMFD_FLAG_SUPPORT_SHARED set, then if slot->userspace_addr != 0, we would validate that the folio slot->userspace_addr points to matches up with the folio guest_memfd would return for the same offset.

Where will the validation be executed? In kvm_gmem_bind()?

Quoted 7 lines

I can think of one way to do this validation, which is to call KVM's hva_to_pfn() function and then call kvm_gmem_get_folio() on the fd and offset, and then check that the PFNs are equal.

However, that would cause the page to be allocated. Any ideas on how we could do this validation without allocating the page?

If the check is in kvm_gmem_bind() and if there's no worry about munmap() and re-mmap() of the shared memory pointed by slot->userspace_addr, maybe below?

mm = kvm->mm;
mmap_read_lock(mm);
vma = vma_lookup(mm, vaddr);
pgoff = ((slot->userspace_addr - vma->vm_start) >> PAGE_SHIFT) + vma->vm_pgoff; mmap_read_unlock(mm);

Then check if pgoff equals to slot->gmem.guest_memfd_offset.

#46 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

In reply to Fuad Tabba
Quoted 28 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

---
 include/linux/kvm_host.h | 15 ++++++-
 include/uapi/linux/kvm.h |  1 +
 virt/kvm/Kconfig         |  5 +++
 virt/kvm/guest_memfd.c   | 92 ++++++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      |  4 ++
 5 files changed, 116 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +0 -0
index 9419fb99f7c2..f3af6bff3232 100644

include/linux/kvm_host.h static inline bool kvm_arch_supports_gmem(struct kvm *kvm) · line 736

Quoted 8 lines
include/linux/kvm_host.h +5 -0
@@ -729,6 +729,17 @@ static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
 }
 #endif
 
+/*
+ * Arch code must define kvm_arch_gmem_supports_shared_mem if support for
+ * private memory is enabled and it supports in-place shared/private conversion.
+ */
+#if !defined(kvm_arch_gmem_supports_shared_mem) && !IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM)

Perhaps the bots already caught this?

I just tried enabling KVM_GMEM_SHARED_MEM on x86 with this patch and it fails with:

|| In file included from arch/x86/kvm/../../../virt/kvm/binary_stats.c:8:
|| ./include/linux/kvm_host.h: In function ‘kvm_mem_from_gmem’:
include/linux/kvm_host.h|2530 col 13| error: implicit declaration of function ‘kvm_arch_gmem_supports_shared_mem’ [-Wimplicit-function-declaration]
||  2530 |         if (kvm_arch_gmem_supports_shared_mem(kvm))
||       |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|| make[4]: *** Waiting for unfinished jobs....

I think the predicate on !CONFIG_KVM_GMEM_SHARED_MEM is wrong.

Shouldn't this always default off? I __think__ this then gets enabled in 11/13?

IOW

include/linux/kvm_host.h +1 -1
index f3af6bff3232..577674e95c09 100644
@@ -733,7 +733,7 @@ static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
  * Arch code must define kvm_arch_gmem_supports_shared_mem if support for
  * private memory is enabled and it supports in-place shared/private conversion.
  */
-#if !defined(kvm_arch_gmem_supports_shared_mem) && !IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM)
+#if !defined(kvm_arch_gmem_supports_shared_mem)
 static inline bool kvm_arch_gmem_supports_shared_mem(struct kvm *kvm)
 {
        return false;
author
#47 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Hi Ira,

Quoted 51 lines · Ira Weiny
Quoted 28 lines · Fuad Tabba

Add support for mmap() and fault() for guest_memfd backed memory in the host for VMs that support in-place conversion between shared and private. To that end, this patch adds the ability to check whether the VM type supports in-place conversion, and only allows mapping its memory if that's the case.

This patch introduces the configuration option KVM_GMEM_SHARED_MEM, which enables support for in-place shared memory.

It also introduces the KVM capability KVM_CAP_GMEM_SHARED_MEM, which indicates that the host can create VMs that support shared memory. Supporting shared memory implies that memory can be mapped when shared with the host.

---
 include/linux/kvm_host.h | 15 ++++++-
 include/uapi/linux/kvm.h |  1 +
 virt/kvm/Kconfig         |  5 +++
 virt/kvm/guest_memfd.c   | 92 ++++++++++++++++++++++++++++++++++++++++
 virt/kvm/kvm_main.c      |  4 ++
 5 files changed, 116 insertions(+), 1 deletion(-)
include/linux/kvm_host.h +0 -0
index 9419fb99f7c2..f3af6bff3232 100644

include/linux/kvm_host.h static inline bool kvm_arch_supports_gmem(struct kvm *kvm) · line 736

Quoted 8 lines
include/linux/kvm_host.h +5 -0
@@ -729,6 +729,17 @@ static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
 }
 #endif
 
+/*
+ * Arch code must define kvm_arch_gmem_supports_shared_mem if support for
+ * private memory is enabled and it supports in-place shared/private conversion.
+ */
+#if !defined(kvm_arch_gmem_supports_shared_mem) && !IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM)

Perhaps the bots already caught this?

I just tried enabling KVM_GMEM_SHARED_MEM on x86 with this patch and it fails with:

|| In file included from arch/x86/kvm/../../../virt/kvm/binary_stats.c:8:
|| ./include/linux/kvm_host.h: In function ‘kvm_mem_from_gmem’:
include/linux/kvm_host.h|2530 col 13| error: implicit declaration of function ‘kvm_arch_gmem_supports_shared_mem’ [-Wimplicit-function-declaration]
||  2530 |         if (kvm_arch_gmem_supports_shared_mem(kvm))
||       |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|| make[4]: *** Waiting for unfinished jobs....
Quoted 4 lines

I think the predicate on !CONFIG_KVM_GMEM_SHARED_MEM is wrong.

Shouldn't this always default off? I __think__ this then gets enabled in 11/13?

You're right. With the other comments from David and Ackerley, this functions is gone, replaced by checking a per-vm flag.

Thanks,
/fuad

Quoted 15 lines

IOW

include/linux/kvm_host.h +1 -1
index f3af6bff3232..577674e95c09 100644
@@ -733,7 +733,7 @@ static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
  * Arch code must define kvm_arch_gmem_supports_shared_mem if support for
  * private memory is enabled and it supports in-place shared/private conversion.
  */
-#if !defined(kvm_arch_gmem_supports_shared_mem) && !IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM)
+#if !defined(kvm_arch_gmem_supports_shared_mem)
 static inline bool kvm_arch_gmem_supports_shared_mem(struct kvm *kvm)
 {
        return false;
#48 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

In reply to Fuad Tabba
kernel test robot noticed the following build errors:

Hi Fuad,

kernel test robot noticed the following build errors:

[auto build test ERROR on b4432656b36e5cc1d50a1f2dc15357543add530e]

url:    https://github.com/intel-lab-lkp/linux/commits/Fuad-Tabba/KVM-Rename-CONFIG_KVM_PRIVATE_MEM-to-CONFIG_KVM_GMEM/20250501-005807
base:   b4432656b36e5cc1d50a1f2dc15357543add530e
patch link:    https://lore.kernel.org/r/20250430165655.605595-9-tabba%40google.com
patch subject: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages
config: powerpc-allyesconfig (https://download.01.org/0day-ci/archive/20250507/[email protected]/config)
compiler: clang version 21.0.0git (https://github.com/llvm/llvm-project f819f46284f2a79790038e1f6649172789734ae8)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250507/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from arch/powerpc/kernel/asm-offsets.c:36:
In file included from arch/powerpc/include/asm/dbell.h:17:
In file included from arch/powerpc/include/asm/kvm_ppc.h:19:

>> include/linux/kvm_host.h:2530:6: error: call to undeclared function 'kvm_arch_gmem_supports_shared_mem'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]

2530 | if (kvm_arch_gmem_supports_shared_mem(kvm))
| ^
1 error generated.
make[3]: *** [scripts/Makefile.build:98: arch/powerpc/kernel/asm-offsets.s] Error 1 make[3]: Target 'prepare' not remade because of errors.
make[2]: *** [Makefile:1282: prepare0] Error 2
make[2]: Target 'prepare' not remade because of errors.
make[1]: *** [Makefile:248: __sub-make] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.

vim +/kvm_arch_gmem_supports_shared_mem +2530 include/linux/kvm_host.h

2527
2528 static inline bool kvm_mem_from_gmem(struct kvm *kvm, gfn_t gfn)
2529 {

> 2530		if (kvm_arch_gmem_supports_shared_mem(kvm))

2531 return true;
2532
2533 return kvm_mem_is_private(kvm, gfn);
2534 }
2535

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#49 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

In reply to Fuad Tabba

virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 397

Quoted 13 lines · Fuad Tabba
virt/kvm/guest_memfd.c +13 -0
+static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
+{
+       struct kvm_gmem *gmem = file->private_data;
+
+       if (!kvm_arch_gmem_supports_shared_mem(gmem->kvm))
+               return -ENODEV;
+
+       if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
+           (VM_SHARED | VM_MAYSHARE)) {
+               return -EINVAL;
+       }
+
+       vm_flags_set(vma, VM_DONTDUMP);

Hi Fuad,

Sorry if I missed this, but why exactly do we set VM_DONTDUMP here? Could you leave a small comment? (I see that it seems to have originally come from Patrick? [1]) I get that guest memory VMAs generally should have VM_DONTDUMP; is there a bigger reason?

[1]: https://lore.kernel.org/kvm/[email protected]/#t

Quoted 4 lines
virt/kvm/guest_memfd.c +4 -0
+       vma->vm_ops = &kvm_gmem_vm_ops;
+
+       return 0;
+}
#50 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Quoted 17 lines · James Houghton

virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 397

Quoted 13 lines · Fuad Tabba
virt/kvm/guest_memfd.c +13 -0
+static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
+{
+       struct kvm_gmem *gmem = file->private_data;
+
+       if (!kvm_arch_gmem_supports_shared_mem(gmem->kvm))
+               return -ENODEV;
+
+       if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
+           (VM_SHARED | VM_MAYSHARE)) {
+               return -EINVAL;
+       }
+
+       vm_flags_set(vma, VM_DONTDUMP);

Hi Fuad,

Quoted 4 lines

Sorry if I missed this, but why exactly do we set VM_DONTDUMP here? Could you leave a small comment? (I see that it seems to have originally come from Patrick? [1]) I get that guest memory VMAs generally should have VM_DONTDUMP; is there a bigger reason?

(David replying)

I assume because we might have inaccessible parts in there that SIGBUS on access.

get_dump_page() does ignore any errors, though (returning NULL), so likely we don't need VM_DONTDUMP.

Signature · 3 lines
Cheers,

David / dhildenb
author
#51 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Hi James.

Quoted 24 lines · David Hildenbrand
Quoted 17 lines · James Houghton

virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 397

Quoted 13 lines · Fuad Tabba
virt/kvm/guest_memfd.c +13 -0
+static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
+{
+       struct kvm_gmem *gmem = file->private_data;
+
+       if (!kvm_arch_gmem_supports_shared_mem(gmem->kvm))
+               return -ENODEV;
+
+       if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
+           (VM_SHARED | VM_MAYSHARE)) {
+               return -EINVAL;
+       }
+
+       vm_flags_set(vma, VM_DONTDUMP);

Hi Fuad,

Quoted 4 lines

Sorry if I missed this, but why exactly do we set VM_DONTDUMP here? Could you leave a small comment? (I see that it seems to have originally come from Patrick? [1]) I get that guest memory VMAs generally should have VM_DONTDUMP; is there a bigger reason?

Quoted 4 lines

(David replying)

I assume because we might have inaccessible parts in there that SIGBUS on access.

That was my thinking.

Quoted 2 lines

get_dump_page() does ignore any errors, though (returning NULL), so likely we don't need VM_DONTDUMP.

In which case I'll remove this from the next respin.

Thanks,
/fuad

Quoted 5 lines

--
Cheers,

David / dhildenb

#52 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

Quoted 33 lines · Fuad Tabba

Hi James.

Quoted 24 lines · David Hildenbrand
Quoted 17 lines · James Houghton

virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 397

Quoted 13 lines · Fuad Tabba
virt/kvm/guest_memfd.c +13 -0
+static int kvm_gmem_mmap(struct file *file, struct vm_area_struct *vma)
+{
+       struct kvm_gmem *gmem = file->private_data;
+
+       if (!kvm_arch_gmem_supports_shared_mem(gmem->kvm))
+               return -ENODEV;
+
+       if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
+           (VM_SHARED | VM_MAYSHARE)) {
+               return -EINVAL;
+       }
+
+       vm_flags_set(vma, VM_DONTDUMP);

Hi Fuad,

Quoted 4 lines

Sorry if I missed this, but why exactly do we set VM_DONTDUMP here? Could you leave a small comment? (I see that it seems to have originally come from Patrick? [1]) I get that guest memory VMAs generally should have VM_DONTDUMP; is there a bigger reason?

Quoted 4 lines

(David replying)

I assume because we might have inaccessible parts in there that SIGBUS on access.

Quoted 6 lines

That was my thinking.

Quoted 2 lines

get_dump_page() does ignore any errors, though (returning NULL), so likely we don't need VM_DONTDUMP.

In which case I'll remove this from the next respin.

SGTM, thanks!

Userspace could remove VM_DONTDUMP by doing MADV_DODUMP, which is why I was curious about this.

And thanks for the extra context[1], Patrick. :)

[1]: https://lore.kernel.org/kvm/[email protected]/

#53 lore raw

Re: [PATCH v8 08/13] KVM: guest_memfd: Allow host to map guest_memfd() pages

In reply to David Hildenbrand
Quoted 11 lines · David Hildenbrand
Quoted 11 lines

virt/kvm/guest_memfd.c static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) · line 397

Quoted 8 lines
               return -ENODEV;
virt/kvm/guest_memfd.c +7 -0
+
+       if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) !=
+           (VM_SHARED | VM_MAYSHARE)) {
+               return -EINVAL;
+       }
+
+       vm_flags_set(vma, VM_DONTDUMP);

Hi Fuad,

Quoted 4 lines
Quoted 4 lines

Sorry if I missed this, but why exactly do we set VM_DONTDUMP here? Could you leave a small comment? (I see that it seems to have originally come from Patrick? [1]) I get that guest memory VMAs generally should have VM_DONTDUMP; is there a bigger reason?

Iirc, I essentially copied my mmap handler from secretmem for that RFC. But even for direct map removal, it seems this is not needed, because get_dump_page goes via GUP, which errors out for direct map removed VMAs. So what David is saying below also applies in that case.

Quoted 12 lines

(David replying)

I assume because we might have inaccessible parts in there that SIGBUS on access.

get_dump_page() does ignore any errors, though (returning NULL), so likely we don't need VM_DONTDUMP.

Signature · 3 lines
Cheers,

David / dhildenb

Best,
Patrick

author
#54 lore raw

PATCH v8 9/13 KVM: arm64: Refactor user_mem_abort() calculation of force_pte

In reply to Fuad Tabba

To simplify the code and to make the assumptions clearer, refactor user_mem_abort() by immediately setting force_pte to true if the conditions are met. Also, remove the comment about logging_active being guaranteed to never be true for VM_PFNMAP memslots, since it's not actually correct.

No functional change intended.

---
 arch/arm64/kvm/mmu.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)
arch/arm64/kvm/mmu.c +4 -9
index 754f2fe0cc67..148a97c129de 100644
@@ -1472,7 +1472,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			  bool fault_is_perm)
 {
 	int ret = 0;
-	bool write_fault, writable, force_pte = false;
+	bool write_fault, writable;
 	bool exec_fault, mte_allowed;
 	bool device = false, vfio_allow_any_uc = false;
 	unsigned long mmu_seq;
@@ -1484,6 +1484,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	gfn_t gfn;
 	kvm_pfn_t pfn;
 	bool logging_active = memslot_is_logging(memslot);
+	bool force_pte = logging_active || is_protected_kvm_enabled();
 	long vma_pagesize, fault_granule;
 	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
 	struct kvm_pgtable *pgt;
@@ -1533,16 +1534,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 		return -EFAULT;
 	}
 
-	/*
-	 * logging_active is guaranteed to never be true for VM_PFNMAP
-	 * memslots.
-	 */
-	if (logging_active || is_protected_kvm_enabled()) {
-		force_pte = true;
+	if (force_pte)
 		vma_shift = PAGE_SHIFT;
-	} else {
+	else
 		vma_shift = get_vma_page_shift(vma, hva);
-	}
 
 	switch (vma_shift) {
 #ifndef __PAGETABLE_PMD_FOLDED
Signature · 1 line
2.49.0.901.g37484f566f-goog
Reviewed-by on v8 9/13
#55 lore raw

Re: [PATCH v8 09/13] KVM: arm64: Refactor user_mem_abort() calculation of force_pte

Quoted 10 lines · Fuad Tabba

To simplify the code and to make the assumptions clearer, refactor user_mem_abort() by immediately setting force_pte to true if the conditions are met. Also, remove the comment about logging_active being guaranteed to never be true for VM_PFNMAP memslots, since it's not actually correct.

No functional change intended.

---

Signature · 3 lines
Cheers,

David / dhildenb
author
#56 lore raw

PATCH v8 10/13 KVM: arm64: Handle guest_memfd()-backed guest page faults

In reply to Fuad Tabba

Add arm64 support for handling guest page faults on guest_memfd backed memslots.

For now, the fault granule is restricted to PAGE_SIZE.

---
 arch/arm64/kvm/mmu.c     | 65 +++++++++++++++++++++++++++-------------
 include/linux/kvm_host.h |  5 ++++
 virt/kvm/kvm_main.c      |  5 ----
 3 files changed, 50 insertions(+), 25 deletions(-)
arch/arm64/kvm/mmu.c +45 -20
index 148a97c129de..d1044c7f78bb 100644
@@ -1466,6 +1466,30 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma)
 	return vma->vm_flags & VM_MTE_ALLOWED;
 }
 
+static kvm_pfn_t faultin_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+			     gfn_t gfn, bool write_fault, bool *writable,
+			     struct page **page, bool is_gmem)
+{
+	kvm_pfn_t pfn;
+	int ret;
+
+	if (!is_gmem)
+		return __kvm_faultin_pfn(slot, gfn, write_fault ? FOLL_WRITE : 0, writable, page);
+
+	*writable = false;
+
+	ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, page, NULL);
+	if (!ret) {
+		*writable = !memslot_is_readonly(slot);
+		return pfn;
+	}
+
+	if (ret == -EHWPOISON)
+		return KVM_PFN_ERR_HWPOISON;
+
+	return KVM_PFN_ERR_NOSLOT_MASK;
+}
+
 static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			  struct kvm_s2_trans *nested,
 			  struct kvm_memory_slot *memslot, unsigned long hva,
@@ -1473,19 +1497,20 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 {
 	int ret = 0;
 	bool write_fault, writable;
-	bool exec_fault, mte_allowed;
+	bool exec_fault, mte_allowed = false;
 	bool device = false, vfio_allow_any_uc = false;
 	unsigned long mmu_seq;
 	phys_addr_t ipa = fault_ipa;
 	struct kvm *kvm = vcpu->kvm;
-	struct vm_area_struct *vma;
+	struct vm_area_struct *vma = NULL;
 	short vma_shift;
 	void *memcache;
-	gfn_t gfn;
+	gfn_t gfn = ipa >> PAGE_SHIFT;
 	kvm_pfn_t pfn;
 	bool logging_active = memslot_is_logging(memslot);
-	bool force_pte = logging_active || is_protected_kvm_enabled();
-	long vma_pagesize, fault_granule;
+	bool is_gmem = kvm_slot_has_gmem(memslot) && kvm_mem_from_gmem(kvm, gfn);
+	bool force_pte = logging_active || is_gmem || is_protected_kvm_enabled();
+	long vma_pagesize, fault_granule = PAGE_SIZE;
 	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
 	struct kvm_pgtable *pgt;
 	struct page *page;
@@ -1522,16 +1547,22 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			return ret;
 	}
 
+	mmap_read_lock(current->mm);
+
 	/*
 	 * Let's check if we will get back a huge page backed by hugetlbfs, or
 	 * get block mapping for device MMIO region.
 	 */
-	mmap_read_lock(current->mm);
-	vma = vma_lookup(current->mm, hva);
-	if (unlikely(!vma)) {
-		kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
-		mmap_read_unlock(current->mm);
-		return -EFAULT;
+	if (!is_gmem) {
+		vma = vma_lookup(current->mm, hva);
+		if (unlikely(!vma)) {
+			kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
+			mmap_read_unlock(current->mm);
+			return -EFAULT;
+		}
+
+		vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
+		mte_allowed = kvm_vma_mte_allowed(vma);
 	}
 
 	if (force_pte)
@@ -1602,18 +1633,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 		ipa &= ~(vma_pagesize - 1);
 	}
 
-	gfn = ipa >> PAGE_SHIFT;
-	mte_allowed = kvm_vma_mte_allowed(vma);
-
-	vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
-
 	/* Don't use the VMA after the unlock -- it may have vanished */
 	vma = NULL;
 
 	/*
 	 * Read mmu_invalidate_seq so that KVM can detect if the results of
-	 * vma_lookup() or __kvm_faultin_pfn() become stale prior to
-	 * acquiring kvm->mmu_lock.
+	 * vma_lookup() or faultin_pfn() become stale prior to acquiring
+	 * kvm->mmu_lock.
 	 *
 	 * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs
 	 * with the smp_wmb() in kvm_mmu_invalidate_end().
@@ -1621,8 +1647,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	mmu_seq = vcpu->kvm->mmu_invalidate_seq;
 	mmap_read_unlock(current->mm);
 
-	pfn = __kvm_faultin_pfn(memslot, gfn, write_fault ? FOLL_WRITE : 0,
-				&writable, &page);
+	pfn = faultin_pfn(kvm, memslot, gfn, write_fault, &writable, &page, is_gmem);
 	if (pfn == KVM_PFN_ERR_HWPOISON) {
 		kvm_send_hwpoison_signal(hva, vma_shift);
 		return 0;
include/linux/kvm_host.h +5 -0
index f3af6bff3232..1b2e4e9a7802 100644
@@ -1882,6 +1882,11 @@ static inline int memslot_id(struct kvm *kvm, gfn_t gfn)
 	return gfn_to_memslot(kvm, gfn)->id;
 }
 
+static inline bool memslot_is_readonly(const struct kvm_memory_slot *slot)
+{
+	return slot->flags & KVM_MEM_READONLY;
+}
+
 static inline gfn_t
 hva_to_gfn_memslot(unsigned long hva, struct kvm_memory_slot *slot)
 {
virt/kvm/kvm_main.c +0 -5
index c75d8e188eb7..d9bca5ba19dc 100644
@@ -2640,11 +2640,6 @@ unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn)
 	return size;
 }
 
-static bool memslot_is_readonly(const struct kvm_memory_slot *slot)
-{
-	return slot->flags & KVM_MEM_READONLY;
-}
-
 static unsigned long __gfn_to_hva_many(const struct kvm_memory_slot *slot, gfn_t gfn,
 				       gfn_t *nr_pages, bool write)
 {
Signature · 1 line
2.49.0.901.g37484f566f-goog
#57 lore raw

Re: [PATCH v8 10/13] KVM: arm64: Handle guest_memfd()-backed guest page faults

Quoted 75 lines · Fuad Tabba

Add arm64 support for handling guest page faults on guest_memfd backed memslots.

For now, the fault granule is restricted to PAGE_SIZE.

---
 arch/arm64/kvm/mmu.c     | 65 +++++++++++++++++++++++++++------------- include/linux/kvm_host.h |  5 ++++
 virt/kvm/kvm_main.c      |  5 ----

 3 files changed, 50 insertions(+), 25 deletions(-)
arch/arm64/kvm/mmu.c +30 -5
index 148a97c129de..d1044c7f78bb 100644
@@ -1466,6 +1466,30 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma)
         return vma->vm_flags & VM_MTE_ALLOWED;
  }
 
+static kvm_pfn_t faultin_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+                            gfn_t gfn, bool write_fault, bool *writable,
+                            struct page **page, bool is_gmem)
+{
+       kvm_pfn_t pfn;
+       int ret;
+
+       if (!is_gmem)
+               return __kvm_faultin_pfn(slot, gfn, write_fault ? FOLL_WRITE : 0, writable, page);
+
+       *writable = false;
+
+       ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, page, NULL);
+       if (!ret) {
+               *writable = !memslot_is_readonly(slot);
+               return pfn;
+       }
+
+       if (ret == -EHWPOISON)
+               return KVM_PFN_ERR_HWPOISON;
+
+       return KVM_PFN_ERR_NOSLOT_MASK;
+}
+
  static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                           struct kvm_s2_trans *nested,
                           struct kvm_memory_slot *memslot, unsigned long hva,
@@ -1473,19 +1497,20 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
  {
         int ret = 0;
         bool write_fault, writable;
-       bool exec_fault, mte_allowed;
+       bool exec_fault, mte_allowed = false;
         bool device = false, vfio_allow_any_uc = false;
         unsigned long mmu_seq;
         phys_addr_t ipa = fault_ipa;
         struct kvm *kvm = vcpu->kvm;
-       struct vm_area_struct *vma;
+       struct vm_area_struct *vma = NULL;
         short vma_shift;
         void *memcache;
-       gfn_t gfn;
+       gfn_t gfn = ipa >> PAGE_SHIFT;
         kvm_pfn_t pfn;
         bool logging_active = memslot_is_logging(memslot);
-       bool force_pte = logging_active || is_protected_kvm_enabled();
-       long vma_pagesize, fault_granule;
+       bool is_gmem = kvm_slot_has_gmem(memslot) && kvm_mem_from_gmem(kvm, gfn);
+       bool force_pte = logging_active || is_gmem || is_protected_kvm_enabled();
+       long vma_pagesize, fault_granule = PAGE_SIZE;
         enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
         struct kvm_pgtable *pgt;
         struct page *page;

arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1550

Quoted 4 lines
arch/arm64/kvm/mmu.c +1 -0
@@ -1522,16 +1547,22 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                         return ret;
         }
 
+       mmap_read_lock(current->mm);

We don't have to take the mmap_lock for gmem faults, right?

I think we should reorganize user_mem_abort() a bit (and I think vma_pagesize and maybe vma_shift should be renamed) given the changes we're making here.

Below is a diff that I think might be a little cleaner. Let me know what you think.

Quoted 46 lines
arch/arm64/kvm/mmu.c +13 -13
+
         /*
          * Let's check if we will get back a huge page backed by hugetlbfs, or
          * get block mapping for device MMIO region.
          */
-       mmap_read_lock(current->mm);
-       vma = vma_lookup(current->mm, hva);
-       if (unlikely(!vma)) {
-               kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
-               mmap_read_unlock(current->mm);
-               return -EFAULT;
+       if (!is_gmem) {
+               vma = vma_lookup(current->mm, hva);
+               if (unlikely(!vma)) {
+                       kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
+                       mmap_read_unlock(current->mm);
+                       return -EFAULT;
+               }
+
+               vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
+               mte_allowed = kvm_vma_mte_allowed(vma);
         }
 
         if (force_pte)
@@ -1602,18 +1633,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                 ipa &= ~(vma_pagesize - 1);
         }
 
-       gfn = ipa >> PAGE_SHIFT;
-       mte_allowed = kvm_vma_mte_allowed(vma);
-
-       vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
-
         /* Don't use the VMA after the unlock -- it may have vanished */
         vma = NULL;
 
         /*
          * Read mmu_invalidate_seq so that KVM can detect if the results of
-        * vma_lookup() or __kvm_faultin_pfn() become stale prior to
-        * acquiring kvm->mmu_lock.
+        * vma_lookup() or faultin_pfn() become stale prior to acquiring
+        * kvm->mmu_lock.
          *
          * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs
          * with the smp_wmb() in kvm_mmu_invalidate_end().

arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1651

Quoted 7 lines
arch/arm64/kvm/mmu.c +1 -2
@@ -1621,8 +1647,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
         mmu_seq = vcpu->kvm->mmu_invalidate_seq;
         mmap_read_unlock(current->mm);
 
-       pfn = __kvm_faultin_pfn(memslot, gfn, write_fault ? FOLL_WRITE : 0,
-                               &writable, &page);
+       pfn = faultin_pfn(kvm, memslot, gfn, write_fault, &writable, &page, is_gmem);
         if (pfn == KVM_PFN_ERR_HWPOISON) {

I think we need to take care to handle HWPOISON properly. I know that it is (or will most likely be) the case that GUP(hva) --> pfn, but with gmem, it *might* not be the case. So the following line isn't right.

I think we need to handle HWPOISON for gmem using memory fault exits instead of sending a SIGBUS to userspace. This would be consistent with how KVM/x86 today handles getting a HWPOISON page back from kvm_gmem_get_pfn(). I'm not entirely sure how KVM/x86 is meant to handle HWPOISON on shared gmem pages yet; I need to keep reading your series.

The reorganization diff below leaves this unfixed.

Quoted 35 lines
arch/arm64/kvm/mmu.c +0 -0
                 kvm_send_hwpoison_signal(hva, vma_shift);
                 return 0;
include/linux/kvm_host.h +5 -0
index f3af6bff3232..1b2e4e9a7802 100644
@@ -1882,6 +1882,11 @@ static inline int memslot_id(struct kvm *kvm, gfn_t gfn)
         return gfn_to_memslot(kvm, gfn)->id;
  }
 
+static inline bool memslot_is_readonly(const struct kvm_memory_slot *slot)
+{
+       return slot->flags & KVM_MEM_READONLY;
+}
+

 static inline gfn_t
 hva_to_gfn_memslot(unsigned long hva, struct kvm_memory_slot *slot) {

virt/kvm/kvm_main.c +0 -6
index c75d8e188eb7..d9bca5ba19dc 100644
@@ -2640,11 +2640,6 @@ unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn)
         return size;
  }
 
-static bool memslot_is_readonly(const struct kvm_memory_slot *slot)
-{
-       return slot->flags & KVM_MEM_READONLY;
-}
-
  static unsigned long __gfn_to_hva_many(const struct kvm_memory_slot *slot, gfn_t gfn,
                                        gfn_t *nr_pages, bool write)
  {
--
Quoted 1 line

2.49.0.901.g37484f566f-goog

Thanks, Fuad! Here's the reorganization/rename diff:

arch/arm64/kvm/mmu.c +58 -62
index d1044c7f78bba..c9eb72fe9013b 100644
@@ -1502,7 +1502,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	unsigned long mmu_seq;
 	phys_addr_t ipa = fault_ipa;
 	struct kvm *kvm = vcpu->kvm;
-	struct vm_area_struct *vma = NULL;
 	short vma_shift;
 	void *memcache;
 	gfn_t gfn = ipa >> PAGE_SHIFT;
@@ -1510,7 +1509,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	bool logging_active = memslot_is_logging(memslot);
 	bool is_gmem = kvm_slot_has_gmem(memslot) && kvm_mem_from_gmem(kvm, gfn);
 	bool force_pte = logging_active || is_gmem || is_protected_kvm_enabled();
-	long vma_pagesize, fault_granule = PAGE_SIZE;
+	long target_size = PAGE_SIZE, fault_granule = PAGE_SIZE;
 	enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
 	struct kvm_pgtable *pgt;
 	struct page *page;
@@ -1547,13 +1546,15 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			return ret;
 	}
 
-	mmap_read_lock(current->mm);
-
 	/*
 	 * Let's check if we will get back a huge page backed by hugetlbfs, or
 	 * get block mapping for device MMIO region.
 	 */
 	if (!is_gmem) {
+		struct vm_area_struct *vma = NULL;
+
+		mmap_read_lock(current->mm);
+
 		vma = vma_lookup(current->mm, hva);
 		if (unlikely(!vma)) {
 			kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
@@ -1563,38 +1564,45 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
 		vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
 		mte_allowed = kvm_vma_mte_allowed(vma);
-	}
-
-	if (force_pte)
-		vma_shift = PAGE_SHIFT;
-	else
-		vma_shift = get_vma_page_shift(vma, hva);
+		vma_shift = force_pte ? get_vma_page_shift(vma, hva) : PAGE_SHIFT;
 
-	switch (vma_shift) {
+		switch (vma_shift) {
 #ifndef __PAGETABLE_PMD_FOLDED
-	case PUD_SHIFT:
-		if (fault_supports_stage2_huge_mapping(memslot, hva, PUD_SIZE))
-			break;
-		fallthrough;
+		case PUD_SHIFT:
+			if (fault_supports_stage2_huge_mapping(memslot, hva, PUD_SIZE))
+				break;
+			fallthrough;
 #endif
-	case CONT_PMD_SHIFT:
-		vma_shift = PMD_SHIFT;
-		fallthrough;
-	case PMD_SHIFT:
-		if (fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE))
+		case CONT_PMD_SHIFT:
+			vma_shift = PMD_SHIFT;
+			fallthrough;
+		case PMD_SHIFT:
+			if (fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE))
+				break;
+			fallthrough;
+		case CONT_PTE_SHIFT:
+			vma_shift = PAGE_SHIFT;
+			force_pte = true;
+			fallthrough;
+		case PAGE_SHIFT:
 			break;
-		fallthrough;
-	case CONT_PTE_SHIFT:
-		vma_shift = PAGE_SHIFT;
-		force_pte = true;
-		fallthrough;
-	case PAGE_SHIFT:
-		break;
-	default:
-		WARN_ONCE(1, "Unknown vma_shift %d", vma_shift);
-	}
+		default:
+			WARN_ONCE(1, "Unknown vma_shift %d", vma_shift);
+		}
 
-	vma_pagesize = 1UL << vma_shift;
+		/*
+		 * Read mmu_invalidate_seq so that KVM can detect if the results of
+		 * vma_lookup() or faultin_pfn() become stale prior to acquiring
+		 * kvm->mmu_lock.
+		 *
+		 * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs
+		 * with the smp_wmb() in kvm_mmu_invalidate_end().
+		 */
+		mmu_seq = vcpu->kvm->mmu_invalidate_seq;
+		mmap_read_unlock(current->mm);
+
+		target_size = 1UL << vma_shift;
+	}
 
 	if (nested) {
 		unsigned long max_map_size;
@@ -1620,7 +1628,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 			max_map_size = PAGE_SIZE;
 
 		force_pte = (max_map_size == PAGE_SIZE);
-		vma_pagesize = min(vma_pagesize, (long)max_map_size);
+		target_size = min(target_size, (long)max_map_size);
 	}
 
 	/*
@@ -1628,27 +1636,15 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	 * ensure we find the right PFN and lay down the mapping in the right
 	 * place.
 	 */
-	if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE) {
-		fault_ipa &= ~(vma_pagesize - 1);
-		ipa &= ~(vma_pagesize - 1);
+	if (target_size == PMD_SIZE || target_size == PUD_SIZE) {
+		fault_ipa &= ~(target_size - 1);
+		ipa &= ~(target_size - 1);
 	}
 
-	/* Don't use the VMA after the unlock -- it may have vanished */
-	vma = NULL;
-
-	/*
-	 * Read mmu_invalidate_seq so that KVM can detect if the results of
-	 * vma_lookup() or faultin_pfn() become stale prior to acquiring
-	 * kvm->mmu_lock.
-	 *
-	 * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs
-	 * with the smp_wmb() in kvm_mmu_invalidate_end().
-	 */
-	mmu_seq = vcpu->kvm->mmu_invalidate_seq;
-	mmap_read_unlock(current->mm);
-
 	pfn = faultin_pfn(kvm, memslot, gfn, write_fault, &writable, &page, is_gmem);
 	if (pfn == KVM_PFN_ERR_HWPOISON) {
+		// TODO: Handle gmem properly. vma_shift
+		// intentionally left uninitialized.
 		kvm_send_hwpoison_signal(hva, vma_shift);
 		return 0;
 	}
@@ -1658,9 +1654,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (kvm_is_device_pfn(pfn)) {
 		/*
 		 * If the page was identified as device early by looking at
-		 * the VMA flags, vma_pagesize is already representing the
+		 * the VMA flags, target_size is already representing the
 		 * largest quantity we can map.  If instead it was mapped
-		 * via __kvm_faultin_pfn(), vma_pagesize is set to PAGE_SIZE
+		 * via __kvm_faultin_pfn(), target_size is set to PAGE_SIZE
 		 * and must not be upgraded.
 		 *
 		 * In both cases, we don't let transparent_hugepage_adjust()
@@ -1699,7 +1695,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
 	kvm_fault_lock(kvm);
 	pgt = vcpu->arch.hw_mmu->pgt;
-	if (mmu_invalidate_retry(kvm, mmu_seq)) {
+	if (!is_gmem && mmu_invalidate_retry(kvm, mmu_seq)) {
 		ret = -EAGAIN;
 		goto out_unlock;
 	}
@@ -1708,16 +1704,16 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	 * If we are not forced to use page mapping, check if we are
 	 * backed by a THP and thus use block mapping if possible.
 	 */
-	if (vma_pagesize == PAGE_SIZE && !(force_pte || device)) {
+	if (target_size == PAGE_SIZE && !(force_pte || device)) {
 		if (fault_is_perm && fault_granule > PAGE_SIZE)
-			vma_pagesize = fault_granule;
-		else
-			vma_pagesize = transparent_hugepage_adjust(kvm, memslot,
+			target_size = fault_granule;
+		else if (!is_gmem)
+			target_size = transparent_hugepage_adjust(kvm, memslot,
 								   hva, &pfn,
 								   &fault_ipa);
 
-		if (vma_pagesize < 0) {
-			ret = vma_pagesize;
+		if (target_size < 0) {
+			ret = target_size;
 			goto out_unlock;
 		}
 	}
@@ -1725,7 +1721,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (!fault_is_perm && !device && kvm_has_mte(kvm)) {
 		/* Check the VMM hasn't introduced a new disallowed VMA */
 		if (mte_allowed) {
-			sanitise_mte_tags(kvm, pfn, vma_pagesize);
+			sanitise_mte_tags(kvm, pfn, target_size);
 		} else {
 			ret = -EFAULT;
 			goto out_unlock;
@@ -1750,10 +1746,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
 	/*
 	 * Under the premise of getting a FSC_PERM fault, we just need to relax
-	 * permissions only if vma_pagesize equals fault_granule. Otherwise,
+	 * permissions only if target_size equals fault_granule. Otherwise,
 	 * kvm_pgtable_stage2_map() should be called to change block size.
 	 */
-	if (fault_is_perm && vma_pagesize == fault_granule) {
+	if (fault_is_perm && target_size == fault_granule) {
 		/*
 		 * Drop the SW bits in favour of those stored in the
 		 * PTE, which will be preserved.
@@ -1761,7 +1757,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 		prot &= ~KVM_NV_GUEST_MAP_SZ;
 		ret = KVM_PGT_FN(kvm_pgtable_stage2_relax_perms)(pgt, fault_ipa, prot, flags);
 	} else {
-		ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, fault_ipa, vma_pagesize,
+		ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, fault_ipa, target_size,
 					     __pfn_to_phys(pfn), prot,
 					     memcache, flags);
 	}
author
#58 lore raw

Re: [PATCH v8 10/13] KVM: arm64: Handle guest_memfd()-backed guest page faults

Hi James,

Quoted 80 lines · James Houghton
Quoted 78 lines · Fuad Tabba

Add arm64 support for handling guest page faults on guest_memfd backed memslots.

For now, the fault granule is restricted to PAGE_SIZE.

---
 arch/arm64/kvm/mmu.c     | 65 +++++++++++++++++++++++++++-------------
 include/linux/kvm_host.h |  5 ++++
 virt/kvm/kvm_main.c      |  5 ----
 3 files changed, 50 insertions(+), 25 deletions(-)
arch/arm64/kvm/mmu.c +30 -5
index 148a97c129de..d1044c7f78bb 100644
@@ -1466,6 +1466,30 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma)
        return vma->vm_flags & VM_MTE_ALLOWED;
 }
 
+static kvm_pfn_t faultin_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
+                            gfn_t gfn, bool write_fault, bool *writable,
+                            struct page **page, bool is_gmem)
+{
+       kvm_pfn_t pfn;
+       int ret;
+
+       if (!is_gmem)
+               return __kvm_faultin_pfn(slot, gfn, write_fault ? FOLL_WRITE : 0, writable, page);
+
+       *writable = false;
+
+       ret = kvm_gmem_get_pfn(kvm, slot, gfn, &pfn, page, NULL);
+       if (!ret) {
+               *writable = !memslot_is_readonly(slot);
+               return pfn;
+       }
+
+       if (ret == -EHWPOISON)
+               return KVM_PFN_ERR_HWPOISON;
+
+       return KVM_PFN_ERR_NOSLOT_MASK;
+}
+
 static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                          struct kvm_s2_trans *nested,
                          struct kvm_memory_slot *memslot, unsigned long hva,
@@ -1473,19 +1497,20 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 {
        int ret = 0;
        bool write_fault, writable;
-       bool exec_fault, mte_allowed;
+       bool exec_fault, mte_allowed = false;
        bool device = false, vfio_allow_any_uc = false;
        unsigned long mmu_seq;
        phys_addr_t ipa = fault_ipa;
        struct kvm *kvm = vcpu->kvm;
-       struct vm_area_struct *vma;
+       struct vm_area_struct *vma = NULL;
        short vma_shift;
        void *memcache;
-       gfn_t gfn;
+       gfn_t gfn = ipa >> PAGE_SHIFT;
        kvm_pfn_t pfn;
        bool logging_active = memslot_is_logging(memslot);
-       bool force_pte = logging_active || is_protected_kvm_enabled();
-       long vma_pagesize, fault_granule;
+       bool is_gmem = kvm_slot_has_gmem(memslot) && kvm_mem_from_gmem(kvm, gfn);
+       bool force_pte = logging_active || is_gmem || is_protected_kvm_enabled();
+       long vma_pagesize, fault_granule = PAGE_SIZE;
        enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
        struct kvm_pgtable *pgt;
        struct page *page;
@@ -1522,16 +1547,22 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                        return ret;
        }
 
Quoted 6 lines

arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1556

Quoted 1 line
arch/arm64/kvm/mmu.c +1 -0
+       mmap_read_lock(current->mm);

We don't have to take the mmap_lock for gmem faults, right?

I think we should reorganize user_mem_abort() a bit (and I think vma_pagesize and maybe vma_shift should be renamed) given the changes we're making here.

Good point.

Quoted 61 lines

Below is a diff that I think might be a little cleaner. Let me know what you think.

Quoted 46 lines
arch/arm64/kvm/mmu.c +13 -13
+
        /*
         * Let's check if we will get back a huge page backed by hugetlbfs, or
         * get block mapping for device MMIO region.
         */
-       mmap_read_lock(current->mm);
-       vma = vma_lookup(current->mm, hva);
-       if (unlikely(!vma)) {
-               kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
-               mmap_read_unlock(current->mm);
-               return -EFAULT;
+       if (!is_gmem) {
+               vma = vma_lookup(current->mm, hva);
+               if (unlikely(!vma)) {
+                       kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
+                       mmap_read_unlock(current->mm);
+                       return -EFAULT;
+               }
+
+               vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
+               mte_allowed = kvm_vma_mte_allowed(vma);
        }
 
        if (force_pte)
@@ -1602,18 +1633,13 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                ipa &= ~(vma_pagesize - 1);
        }
 
-       gfn = ipa >> PAGE_SHIFT;
-       mte_allowed = kvm_vma_mte_allowed(vma);
-
-       vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
-
        /* Don't use the VMA after the unlock -- it may have vanished */
        vma = NULL;
 
        /*
         * Read mmu_invalidate_seq so that KVM can detect if the results of
-        * vma_lookup() or __kvm_faultin_pfn() become stale prior to
-        * acquiring kvm->mmu_lock.
+        * vma_lookup() or faultin_pfn() become stale prior to acquiring
+        * kvm->mmu_lock.
         *
         * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs
         * with the smp_wmb() in kvm_mmu_invalidate_end().

arch/arm64/kvm/mmu.c static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, · line 1651

Quoted 7 lines
arch/arm64/kvm/mmu.c +1 -2
@@ -1621,8 +1647,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        mmu_seq = vcpu->kvm->mmu_invalidate_seq;
        mmap_read_unlock(current->mm);
 
-       pfn = __kvm_faultin_pfn(memslot, gfn, write_fault ? FOLL_WRITE : 0,
-                               &writable, &page);
+       pfn = faultin_pfn(kvm, memslot, gfn, write_fault, &writable, &page, is_gmem);
        if (pfn == KVM_PFN_ERR_HWPOISON) {

I think we need to take care to handle HWPOISON properly. I know that it is (or will most likely be) the case that GUP(hva) --> pfn, but with gmem, it *might* not be the case. So the following line isn't right.

Quoted 5 lines

I think we need to handle HWPOISON for gmem using memory fault exits instead of sending a SIGBUS to userspace. This would be consistent with how KVM/x86 today handles getting a HWPOISON page back from kvm_gmem_get_pfn(). I'm not entirely sure how KVM/x86 is meant to handle HWPOISON on shared gmem pages yet; I need to keep reading your series.

You're right. In the next respin (coming soon), Ackerley has added a patch that performs a best-effort check to ensure that hva matches the gfn.

Quoted 28 lines

The reorganization diff below leaves this unfixed.

Quoted 26 lines
arch/arm64/kvm/mmu.c +0 -0
                kvm_send_hwpoison_signal(hva, vma_shift);
                return 0;
include/linux/kvm_host.h +5 -0
index f3af6bff3232..1b2e4e9a7802 100644
@@ -1882,6 +1882,11 @@ static inline int memslot_id(struct kvm *kvm, gfn_t gfn)
        return gfn_to_memslot(kvm, gfn)->id;
 }
 
+static inline bool memslot_is_readonly(const struct kvm_memory_slot *slot)
+{
+       return slot->flags & KVM_MEM_READONLY;
+}
+
 static inline gfn_t
 hva_to_gfn_memslot(unsigned long hva, struct kvm_memory_slot *slot)
 {
virt/kvm/kvm_main.c +0 -0
index c75d8e188eb7..d9bca5ba19dc 100644
@@ -2640,11 +2640,6 @@ unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn)
        return size;
 }
 
Quoted 12 lines
Quoted 10 lines
virt/kvm/kvm_main.c +0 -6
-static bool memslot_is_readonly(const struct kvm_memory_slot *slot)
-{
-       return slot->flags & KVM_MEM_READONLY;
-}
-
 static unsigned long __gfn_to_hva_many(const struct kvm_memory_slot *slot, gfn_t gfn,
                                       gfn_t *nr_pages, bool write)
 {
--

2.49.0.901.g37484f566f-goog

Thanks, Fuad! Here's the reorganization/rename diff:

Thank you James. This is very helpful.

Cheers,
/fuad

Quoted 230 lines
arch/arm64/kvm/mmu.c +58 -62
index d1044c7f78bba..c9eb72fe9013b 100644
@@ -1502,7 +1502,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        unsigned long mmu_seq;
        phys_addr_t ipa = fault_ipa;
        struct kvm *kvm = vcpu->kvm;
-       struct vm_area_struct *vma = NULL;
        short vma_shift;
        void *memcache;
        gfn_t gfn = ipa >> PAGE_SHIFT;
@@ -1510,7 +1509,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        bool logging_active = memslot_is_logging(memslot);
        bool is_gmem = kvm_slot_has_gmem(memslot) && kvm_mem_from_gmem(kvm, gfn);
        bool force_pte = logging_active || is_gmem || is_protected_kvm_enabled();
-       long vma_pagesize, fault_granule = PAGE_SIZE;
+       long target_size = PAGE_SIZE, fault_granule = PAGE_SIZE;
        enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R;
        struct kvm_pgtable *pgt;
        struct page *page;
@@ -1547,13 +1546,15 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                        return ret;
        }
 
-       mmap_read_lock(current->mm);
-
        /*
         * Let's check if we will get back a huge page backed by hugetlbfs, or
         * get block mapping for device MMIO region.
         */
        if (!is_gmem) {
+               struct vm_area_struct *vma = NULL;
+
+               mmap_read_lock(current->mm);
+
                vma = vma_lookup(current->mm, hva);
                if (unlikely(!vma)) {
                        kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
@@ -1563,38 +1564,45 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
                vfio_allow_any_uc = vma->vm_flags & VM_ALLOW_ANY_UNCACHED;
                mte_allowed = kvm_vma_mte_allowed(vma);
-       }
-
-       if (force_pte)
-               vma_shift = PAGE_SHIFT;
-       else
-               vma_shift = get_vma_page_shift(vma, hva);
+               vma_shift = force_pte ? get_vma_page_shift(vma, hva) : PAGE_SHIFT;
 
-       switch (vma_shift) {
+               switch (vma_shift) {
 #ifndef __PAGETABLE_PMD_FOLDED
-       case PUD_SHIFT:
-               if (fault_supports_stage2_huge_mapping(memslot, hva, PUD_SIZE))
-                       break;
-               fallthrough;
+               case PUD_SHIFT:
+                       if (fault_supports_stage2_huge_mapping(memslot, hva, PUD_SIZE))
+                               break;
+                       fallthrough;
 #endif
-       case CONT_PMD_SHIFT:
-               vma_shift = PMD_SHIFT;
-               fallthrough;
-       case PMD_SHIFT:
-               if (fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE))
+               case CONT_PMD_SHIFT:
+                       vma_shift = PMD_SHIFT;
+                       fallthrough;
+               case PMD_SHIFT:
+                       if (fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE))
+                               break;
+                       fallthrough;
+               case CONT_PTE_SHIFT:
+                       vma_shift = PAGE_SHIFT;
+                       force_pte = true;
+                       fallthrough;
+               case PAGE_SHIFT:
                        break;
-               fallthrough;
-       case CONT_PTE_SHIFT:
-               vma_shift = PAGE_SHIFT;
-               force_pte = true;
-               fallthrough;
-       case PAGE_SHIFT:
-               break;
-       default:
-               WARN_ONCE(1, "Unknown vma_shift %d", vma_shift);
-       }
+               default:
+                       WARN_ONCE(1, "Unknown vma_shift %d", vma_shift);
+               }
 
-       vma_pagesize = 1UL << vma_shift;
+               /*
+                * Read mmu_invalidate_seq so that KVM can detect if the results of
+                * vma_lookup() or faultin_pfn() become stale prior to acquiring
+                * kvm->mmu_lock.
+                *
+                * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs
+                * with the smp_wmb() in kvm_mmu_invalidate_end().
+                */
+               mmu_seq = vcpu->kvm->mmu_invalidate_seq;
+               mmap_read_unlock(current->mm);
+
+               target_size = 1UL << vma_shift;
+       }
 
        if (nested) {
                unsigned long max_map_size;
@@ -1620,7 +1628,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                        max_map_size = PAGE_SIZE;
 
                force_pte = (max_map_size == PAGE_SIZE);
-               vma_pagesize = min(vma_pagesize, (long)max_map_size);
+               target_size = min(target_size, (long)max_map_size);
        }
 
        /*
@@ -1628,27 +1636,15 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
         * ensure we find the right PFN and lay down the mapping in the right
         * place.
         */
-       if (vma_pagesize == PMD_SIZE || vma_pagesize == PUD_SIZE) {
-               fault_ipa &= ~(vma_pagesize - 1);
-               ipa &= ~(vma_pagesize - 1);
+       if (target_size == PMD_SIZE || target_size == PUD_SIZE) {
+               fault_ipa &= ~(target_size - 1);
+               ipa &= ~(target_size - 1);
        }
 
-       /* Don't use the VMA after the unlock -- it may have vanished */
-       vma = NULL;
-
-       /*
-        * Read mmu_invalidate_seq so that KVM can detect if the results of
-        * vma_lookup() or faultin_pfn() become stale prior to acquiring
-        * kvm->mmu_lock.
-        *
-        * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs
-        * with the smp_wmb() in kvm_mmu_invalidate_end().
-        */
-       mmu_seq = vcpu->kvm->mmu_invalidate_seq;
-       mmap_read_unlock(current->mm);
-
        pfn = faultin_pfn(kvm, memslot, gfn, write_fault, &writable, &page, is_gmem);
        if (pfn == KVM_PFN_ERR_HWPOISON) {
+               // TODO: Handle gmem properly. vma_shift
+               // intentionally left uninitialized.
                kvm_send_hwpoison_signal(hva, vma_shift);
                return 0;
        }
@@ -1658,9 +1654,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        if (kvm_is_device_pfn(pfn)) {
                /*
                 * If the page was identified as device early by looking at
-                * the VMA flags, vma_pagesize is already representing the
+                * the VMA flags, target_size is already representing the
                 * largest quantity we can map.  If instead it was mapped
-                * via __kvm_faultin_pfn(), vma_pagesize is set to PAGE_SIZE
+                * via __kvm_faultin_pfn(), target_size is set to PAGE_SIZE
                 * and must not be upgraded.
                 *
                 * In both cases, we don't let transparent_hugepage_adjust()
@@ -1699,7 +1695,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
        kvm_fault_lock(kvm);
        pgt = vcpu->arch.hw_mmu->pgt;
-       if (mmu_invalidate_retry(kvm, mmu_seq)) {
+       if (!is_gmem && mmu_invalidate_retry(kvm, mmu_seq)) {
                ret = -EAGAIN;
                goto out_unlock;
        }
@@ -1708,16 +1704,16 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
         * If we are not forced to use page mapping, check if we are
         * backed by a THP and thus use block mapping if possible.
         */
-       if (vma_pagesize == PAGE_SIZE && !(force_pte || device)) {
+       if (target_size == PAGE_SIZE && !(force_pte || device)) {
                if (fault_is_perm && fault_granule > PAGE_SIZE)
-                       vma_pagesize = fault_granule;
-               else
-                       vma_pagesize = transparent_hugepage_adjust(kvm, memslot,
+                       target_size = fault_granule;
+               else if (!is_gmem)
+                       target_size = transparent_hugepage_adjust(kvm, memslot,
                                                                   hva, &pfn,
                                                                   &fault_ipa);
 
-               if (vma_pagesize < 0) {
-                       ret = vma_pagesize;
+               if (target_size < 0) {
+                       ret = target_size;
                        goto out_unlock;
                }
        }
@@ -1725,7 +1721,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
        if (!fault_is_perm && !device && kvm_has_mte(kvm)) {
                /* Check the VMM hasn't introduced a new disallowed VMA */
                if (mte_allowed) {
-                       sanitise_mte_tags(kvm, pfn, vma_pagesize);
+                       sanitise_mte_tags(kvm, pfn, target_size);
                } else {
                        ret = -EFAULT;
                        goto out_unlock;
@@ -1750,10 +1746,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
        /*
         * Under the premise of getting a FSC_PERM fault, we just need to relax
-        * permissions only if vma_pagesize equals fault_granule. Otherwise,
+        * permissions only if target_size equals fault_granule. Otherwise,
         * kvm_pgtable_stage2_map() should be called to change block size.
         */
-       if (fault_is_perm && vma_pagesize == fault_granule) {
+       if (fault_is_perm && target_size == fault_granule) {
                /*
                 * Drop the SW bits in favour of those stored in the
                 * PTE, which will be preserved.
@@ -1761,7 +1757,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
                prot &= ~KVM_NV_GUEST_MAP_SZ;
                ret = KVM_PGT_FN(kvm_pgtable_stage2_relax_perms)(pgt, fault_ipa, prot, flags);
        } else {
-               ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, fault_ipa, vma_pagesize,
+               ret = KVM_PGT_FN(kvm_pgtable_stage2_map)(pgt, fault_ipa, target_size,
                                             __pfn_to_phys(pfn), prot,
                                             memcache, flags);
        }
author
#59 lore raw

PATCH v8 11/13 KVM: arm64: Enable mapping guest_memfd in arm64

In reply to Fuad Tabba

Enable mapping guest_memfd in arm64. For now, it applies to all VMs in arm64 that use guest_memfd. In the future, new VM types can restrict this via kvm_arch_gmem_supports_shared_mem().

---
 arch/arm64/include/asm/kvm_host.h | 12 ++++++++++++
 arch/arm64/kvm/Kconfig            |  1 +
 2 files changed, 13 insertions(+)
arch/arm64/include/asm/kvm_host.h +12 -0
index 08ba91e6fb03..1b1753e8021a 100644
@@ -1593,4 +1593,16 @@ static inline bool kvm_arch_has_irq_bypass(void)
 	return true;
 }
 
+#ifdef CONFIG_KVM_GMEM
+static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
+{
+	return IS_ENABLED(CONFIG_KVM_GMEM);
+}
+
+static inline bool kvm_arch_gmem_supports_shared_mem(struct kvm *kvm)
+{
+	return IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM);
+}
+#endif /* CONFIG_KVM_GMEM */
+
 #endif /* __ARM64_KVM_HOST_H__ */
arch/arm64/kvm/Kconfig +1 -0
index 096e45acadb2..8c1e1964b46a 100644
@@ -38,6 +38,7 @@ menuconfig KVM
 	select HAVE_KVM_VCPU_RUN_PID_CHANGE
 	select SCHED_INFO
 	select GUEST_PERF_EVENTS if PERF_EVENTS
+	select KVM_GMEM_SHARED_MEM
 	help
 	  Support hosting virtualized guest machines.
 
Signature · 1 line
2.49.0.901.g37484f566f-goog
#60 lore raw

Re: [PATCH v8 11/13] KVM: arm64: Enable mapping guest_memfd in arm64

arch/arm64/include/asm/kvm_host.h static inline bool kvm_arch_has_irq_bypass(void) · line 1599

Quoted 4 lines · Fuad Tabba
arch/arm64/include/asm/kvm_host.h +4 -0
+#ifdef CONFIG_KVM_GMEM
+static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
+{
+       return IS_ENABLED(CONFIG_KVM_GMEM);

How about just return true;? :)

author
#61 lore raw

Re: [PATCH v8 11/13] KVM: arm64: Enable mapping guest_memfd in arm64

Quoted 8 lines · James Houghton

arch/arm64/include/asm/kvm_host.h static inline bool kvm_arch_has_irq_bypass(void) · line 1599

Quoted 4 lines · Fuad Tabba
arch/arm64/include/asm/kvm_host.h +4 -0
+#ifdef CONFIG_KVM_GMEM
+static inline bool kvm_arch_supports_gmem(struct kvm *kvm)
+{
+       return IS_ENABLED(CONFIG_KVM_GMEM);

How about just return true;? :)

Ack.

Thanks!
/fuad

author
#62 lore raw

PATCH v8 12/13 KVM: x86: KVM_X86_SW_PROTECTED_VM to support guest_memfd shared memory

In reply to Fuad Tabba

The KVM_X86_SW_PROTECTED_VM type is meant for experimentation and does not have underlying support for protected guests. This makes it a good candidate for testing mapping shared memory. Therefore, only when the kconfig option for in-place shared memory is enabled (KVM_GMEM_SHARED_MEM), mark this type as supporting shared memory.

This means that this memory is considered by guest_memfd to be shared with the host, which is now able to map and fault in guest_memfd memory belonging to this VM type.

---
 arch/x86/include/asm/kvm_host.h | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/include/asm/kvm_host.h +5 -0
index 709cc2a7ba66..1858dde449c3 100644
@@ -2255,8 +2255,13 @@ void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level,
 
 #ifdef CONFIG_KVM_GMEM
 #define kvm_arch_supports_gmem(kvm) ((kvm)->arch.supports_gmem)
+
+#define kvm_arch_gmem_supports_shared_mem(kvm)         \
+	(IS_ENABLED(CONFIG_KVM_GMEM_SHARED_MEM) &&      \
+	 ((kvm)->arch.vm_type == KVM_X86_SW_PROTECTED_VM))
 #else
 #define kvm_arch_supports_gmem(kvm) false
+#define kvm_arch_gmem_supports_shared_mem(kvm) false
 #endif
 
 #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state)
Signature · 1 line
2.49.0.901.g37484f566f-goog
author
#63 lore raw

PATCH v8 13/13 KVM: guest_memfd: selftests: guest_memfd mmap() test when mapping is allowed

In reply to Fuad Tabba

Expand the guest_memfd selftests to include testing mapping guest memory for VM types that support it.

Also, build the guest_memfd selftest for arm64.

---
 tools/testing/selftests/kvm/Makefile.kvm      |  1 +
 .../testing/selftests/kvm/guest_memfd_test.c  | 75 +++++++++++++++++--
 2 files changed, 70 insertions(+), 6 deletions(-)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index f62b0a5aba35..ccf95ed037c3 100644
@@ -163,6 +163,7 @@ TEST_GEN_PROGS_arm64 += access_tracking_perf_test
 TEST_GEN_PROGS_arm64 += arch_timer
 TEST_GEN_PROGS_arm64 += coalesced_io_test
 TEST_GEN_PROGS_arm64 += dirty_log_perf_test
+TEST_GEN_PROGS_arm64 += guest_memfd_test
 TEST_GEN_PROGS_arm64 += get-reg-list
 TEST_GEN_PROGS_arm64 += memslot_modification_stress_test
 TEST_GEN_PROGS_arm64 += memslot_perf_test
tools/testing/selftests/kvm/guest_memfd_test.c +69 -6

110 lines not on this page · raw message

Signature · 1 line
2.49.0.901.g37484f566f-goog