Skip to contentSkip to conversation

KVM: x86: Introduce memory protection attributes

PATCH Superseded Paolo Bonzini 3 participants 43 messages Sep 18 – Sep 21

Subsystems (10)
Stable branch Documentation x86 KVM +6 more subsystems

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Paolo Bonzini

author maintainer
#0 lore raw

v3 cover KVM: x86: Introduce memory protection attributes

[cover letter copied from v2 - sorry for the very quick turnaround but it's pointless to ask for reviews with the issues reported by sashiko]

This series introduces a mechanism to let userspace block read, write or execute access to individual GFNs via KVM's memory attribute mechanism, and have them reported via KVM_EXIT_MEMORY_FAULT. It is mostly the work of Nicolas Saenz Julienne, with my working consisting in reorganization, code cleanup, and using the recently revamped MMU code (ACC_* masks and kvm_page_format).

KVM needs to check the attributes anytime KVM takes GPAs as input for any action initiated by the guest; if the memory attributes are incompatible with such action, it should be stopped. This means that there are quite a lot of cases to handle. While some families of functions can be handled in one step, there are simply many places that do memory access.

Along the way, the patches fix some issues in the memory attributes code, that surfaced due to having more than one attribute.

Paolo

v2->v3 is just a bunch of sashiko fixes:
- new patch "KVM: x86/hyperv: do not overwrite hc->ingpa for slow SIGNAL_EVENT hypercall"
- rewritten "KVM: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING" to handle cross-patch accesses
- rewrite choice of KVM_FILTER_SHARED/KVM_FILTER_PRIVATE prior to setting memory attributes; even though the previous version worked, it relied on not having any attribute at all for has_private_mem VMs
- fix WARN_ON_ONCE/WARN_ONCE confusion
- fix loongarch compilation
- new (not really satisfactory_ patch "KVM: Take memory protections into account for __kvm_vcpu_map"
- fix incorrect NOT in "KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory attributes"
- do not use kvm_mmu_prepare_memory_fault_exit() for attribute exits on PTEs
- new patch "KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock"
- testcase fixes

Anish Moorthy (1):
KVM: Define and communicate KVM_EXIT_MEMORY_FAULT RWX flags to
userspace

Nicolas Saenz Julienne (12):
KVM: selftests: Take into account mixed memory fault flags
KVM: x86/mmu: Init memslot hugepage information for non-private_mem
VMs too
KVM: Introduce NR/NW/NX memory attributes
KVM: Include memory protections in result of gfn->hva conversion
KVM: Take memory protections into account for memory read/write/fetch
KVM: Encapsulate memattrs array into anonymous struct
KVM: Introduce a generation number for memory attributes
KVM: Take memory protections into account for accesses with cached

    gfn->hva

KVM: pfncache: Fail to refresh if it contains memory protections
KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory
attributes
KVM: x86/mmu: Take memory protection attributes into account during
faults
KVM: x86/mmu: Issue memory fault exit if walk failed due to memory
attribute

Paolo Bonzini (18):
KVM: x86/hyperv: do not overwrite hc->ingpa for slow SIGNAL_EVENT
hypercall
KVM: selftests: Test address translation for Hyper-V direct L2
hypercalls
KVM: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING
KVM: x86: Introduce memory fault on invalid hypercalls reads/writes
KVM: selftests: test hypercall memory fault exits
KVM: x86/mmu: intersect writability from __kvm_faultin_pfn with

    fault->map_writable

KVM: x86/mmu: Extend map_writable to a full ACC_* mask
KVM: pass kvm == NULL case to kvm_arch_has_private_mem
KVM: adjust for presence of more than one attribute
KVM: Introduce kvm_fetch_guest_page() and use it for x86
KVM: Take memory protections into account for __kvm_vcpu_map
KVM: loongarch: do full validity check on the gfn-to-hva cache
KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()
KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock
KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs
KVM: x86: selftests: Introduce memory protection attributes test
KVM: x86: selftests: Introduce memory attributes PTE test
KVM: x86: selftests: Introduce memory attributes side-channel tests

 Documentation/virt/kvm/api.rst                |  38 +-
 arch/loongarch/kvm/vcpu.c                     |  12 +-
 arch/x86/include/asm/kvm_host.h               |   4 +-
 arch/x86/kvm/Kconfig                          |   4 +-
 arch/x86/kvm/hyperv.c                         | 172 +++++--
 arch/x86/kvm/mmu/mmu.c                        | 183 +++++--
 arch/x86/kvm/mmu/mmu_internal.h               |  21 +-
 arch/x86/kvm/mmu/mmutrace.h                   |  36 ++
 arch/x86/kvm/mmu/paging_tmpl.h                |  25 +-
 arch/x86/kvm/mmu/spte.c                       |  12 +-
 arch/x86/kvm/mmu/spte.h                       |  13 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    |   2 +-
 arch/x86/kvm/x86.c                            |  66 ++-
 include/linux/kvm_host.h                      | 128 ++++-
 include/linux/kvm_types.h                     |   6 +-
 include/trace/events/kvm.h                    |  14 +-
 include/uapi/linux/kvm.h                      |   7 +
 tools/include/uapi/linux/kvm.h                |   3 +
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../testing/selftests/kvm/include/kvm_util.h  |  32 +-
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../testing/selftests/kvm/lib/x86/processor.c |   5 +
 .../testing/selftests/kvm/memory_attributes.c | 453 ++++++++++++++++++
 tools/testing/selftests/kvm/x86/hcall_fault.c | 246 ++++++++++
 .../testing/selftests/kvm/x86/hyperv_evmcs.c  |  16 +-
 .../selftests/kvm/x86/hyperv_svm_test.c       |  15 +-
 .../selftests/kvm/x86/memory_attributes.c     | 415 ++++++++++++++++
 .../kvm/x86/private_mem_kvm_exits_test.c      |   6 +-
 virt/kvm/kvm_main.c                           | 271 +++++++++--
 virt/kvm/pfncache.c                           |  30 +-
 30 files changed, 2027 insertions(+), 211 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/memory_attributes.c
 create mode 100644 tools/testing/selftests/kvm/x86/hcall_fault.c
 create mode 100644 tools/testing/selftests/kvm/x86/memory_attributes.c
Signature · 1 line
2.52.0
author maintainer
#1 lore raw

PATCH 1/31 KVM: x86/hyperv: do not overwrite hc->ingpa for slow SIGNAL_EVENT hypercall

When a guest makes a slow HVCALL_SIGNAL_EVENT hypercall with a connection ID that is valid in userspace but not registered in the kernel conn_to_evt, kvm_hvcall_signal_event() reads the connection ID and overwrites hc->ingpa. However, hc->param still signals that the hypercall was a slow one, and userspace will then treat the connection ID as an address.

Fixes bd38b32053eb ("KVM: hyper-v: Collect hypercall params into struct")
---
 arch/x86/kvm/hyperv.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/x86/kvm/hyperv.c +9 -6
index 604651cb2739..d2921d443fde 100644
@@ -2436,19 +2436,22 @@ static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *h
 {
 	struct kvm_hv *hv = to_kvm_hv(vcpu->kvm);
 	struct eventfd_ctx *eventfd;
+	u64 conn_id;
 
 	if (unlikely(!hc->fast)) {
 		int ret;
 		gpa_t gpa = hc->ingpa;
 
-		if ((gpa & (__alignof__(hc->ingpa) - 1)) ||
-		    offset_in_page(gpa) + sizeof(hc->ingpa) > PAGE_SIZE)
+		if ((gpa & (__alignof__(conn_id) - 1)) ||
+		    offset_in_page(gpa) + sizeof(conn_id) > PAGE_SIZE)
 			return HV_STATUS_INVALID_ALIGNMENT;
 
 		ret = kvm_vcpu_read_guest(vcpu, gpa,
-					  &hc->ingpa, sizeof(hc->ingpa));
+					  &conn_id, sizeof(conn_id));
 		if (ret < 0)
 			return HV_STATUS_INVALID_ALIGNMENT;
+	} else {
+		conn_id = hc->ingpa;
 	}
 
 	/*
@@ -2456,15 +2459,15 @@ static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *h
 	 * have no use for it, and in all known usecases it is zero, so just
 	 * report lookup failure if it isn't.
 	 */
-	if (hc->ingpa & 0xffff00000000ULL)
+	if (conn_id & 0xffff00000000ULL)
 		return HV_STATUS_INVALID_PORT_ID;
 	/* remaining bits are reserved-zero */
-	if (hc->ingpa & ~KVM_HYPERV_CONN_ID_MASK)
+	if (conn_id & ~KVM_HYPERV_CONN_ID_MASK)
 		return HV_STATUS_INVALID_HYPERCALL_INPUT;
 
 	/* the eventfd is protected by vcpu->kvm->srcu, but conn_to_evt isn't */
 	rcu_read_lock();
-	eventfd = idr_find(&hv->conn_to_evt, hc->ingpa);
+	eventfd = idr_find(&hv->conn_to_evt, conn_id);
 	rcu_read_unlock();
 	if (!eventfd)
 		return HV_STATUS_INVALID_PORT_ID;
Signature · 1 line
2.52.0
author maintainer
#2 lore raw

PATCH 2/31 KVM: selftests: Take into account mixed memory fault flags

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Update x86_64/private_mem_kvm_exits_test to take into account memory fault flags might contain multiple bits set while remaining valid---for example KVM might return KVM_MEMORY_EXIT_FLAG_READ in addition to KVM_MEMORY_EXIT_FLAG_PRIVATE.

---
 .../testing/selftests/kvm/x86/private_mem_kvm_exits_test.c  | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c +4 -2
index e6d16e9d6120..33fd73465432 100644
@@ -74,7 +74,8 @@ static void test_private_access_memslot_deleted(void)
 	exit_reason = (u32)(u64)thread_return;
 
 	TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MEMORY_FAULT);
-	TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, KVM_MEMORY_EXIT_FLAG_PRIVATE);
+	TEST_ASSERT(vcpu->run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_PRIVATE,
+		    "Memory fault didn't occur on a private memory access");
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, EXITS_TEST_GPA);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, EXITS_TEST_SIZE);
 
@@ -103,7 +104,8 @@ static void test_private_access_memslot_not_private(void)
 	exit_reason = run_vcpu_get_exit_reason(vcpu);
 
 	TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MEMORY_FAULT);
-	TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, KVM_MEMORY_EXIT_FLAG_PRIVATE);
+	TEST_ASSERT(vcpu->run->memory_fault.flags & KVM_MEMORY_EXIT_FLAG_PRIVATE,
+		    "Memory fault didn't occur on a private memory access");
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, EXITS_TEST_GPA);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, EXITS_TEST_SIZE);
 
Signature · 1 line
2.52.0
author maintainer
#3 lore raw

PATCH 3/31 KVM: Define and communicate KVM_EXIT_MEMORY_FAULT RWX flags to userspace

In reply to Paolo Bonzini

From: Anish Moorthy <[email protected]>

kvm_prepare_memory_fault_exit() already takes parameters describing the RWX-ness of the relevant access but doesn't actually do anything with them. Define and use the flags necessary to pass this information on to userspace.

---
 Documentation/virt/kvm/api.rst | 5 +++++
 include/linux/kvm_host.h       | 9 ++++++++-
 include/uapi/linux/kvm.h       | 3 +++
 3 files changed, 16 insertions(+), 1 deletion(-)
Documentation/virt/kvm/api.rst +5 -0
index 4eb7e75a7473..37028ce019e3 100644
@@ -7279,6 +7279,9 @@ spec refer, https://github.com/riscv/riscv-sbi-doc.
 
 		/* KVM_EXIT_MEMORY_FAULT */
 		struct {
+  #define KVM_MEMORY_EXIT_FLAG_READ     (1ULL << 0)
+  #define KVM_MEMORY_EXIT_FLAG_WRITE    (1ULL << 1)
+  #define KVM_MEMORY_EXIT_FLAG_EXEC     (1ULL << 2)
   #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
 			__u64 flags;
 			__u64 gpa;
@@ -7290,6 +7293,8 @@ could not be resolved by KVM.  The 'gpa' and 'size' (in bytes) describe the
 guest physical address range [gpa, gpa + size) of the fault.  The 'flags' field
 describes properties of the faulting access that are likely pertinent:
 
+ - KVM_MEMORY_EXIT_FLAG_READ/WRITE/EXEC - When set, indicates that the memory
+   fault occurred on a read/write/exec access respectively.
  - KVM_MEMORY_EXIT_FLAG_PRIVATE - When set, indicates the memory fault occurred
    on a private memory access.  When clear, indicates the fault occurred on a
    shared access.
include/linux/kvm_host.h +8 -1
index 03bfc92864b6..ce5757e85ae3 100644
@@ -2546,8 +2546,15 @@ static inline void kvm_prepare_memory_fault_exit(struct kvm_vcpu *vcpu,
 	vcpu->run->memory_fault.gpa = gpa;
 	vcpu->run->memory_fault.size = size;
 
-	/* RWX flags are not (yet) defined or communicated to userspace. */
 	vcpu->run->memory_fault.flags = 0;
+
+	if (is_write)
+		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_WRITE;
+	else if (is_exec)
+		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_EXEC;
+	else
+		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_READ;
+
 	if (is_private)
 		vcpu->run->memory_fault.flags |= KVM_MEMORY_EXIT_FLAG_PRIVATE;
 }
include/uapi/linux/kvm.h +3 -0
index ac2d77d14963..15a3090f067e 100644
@@ -457,6 +457,9 @@ struct kvm_run {
 		} notify;
 		/* KVM_EXIT_MEMORY_FAULT */
 		struct {
+#define KVM_MEMORY_EXIT_FLAG_READ       (1ULL << 0)
+#define KVM_MEMORY_EXIT_FLAG_WRITE      (1ULL << 1)
+#define KVM_MEMORY_EXIT_FLAG_EXEC       (1ULL << 2)
 #define KVM_MEMORY_EXIT_FLAG_PRIVATE	(1ULL << 3)
 			__u64 flags;
 			__u64 gpa;
Signature · 1 line
2.52.0
author maintainer
#4 lore raw

PATCH 4/31 KVM: selftests: Test address translation for Hyper-V direct L2 hypercalls

In reply to Paolo Bonzini

Test that the TLB flush hypercalls, when executed by L2, translate their input addresses through the guest's EPT or NPT.

---
 tools/testing/selftests/kvm/x86/hyperv_evmcs.c    | 12 +++++++++++-
 tools/testing/selftests/kvm/x86/hyperv_svm_test.c | 13 ++++++++++++-
 2 files changed, 23 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/x86/hyperv_evmcs.c +11 -1
index 63ea1533e4ea..29baae74ad3a 100644
@@ -18,6 +18,8 @@
 #include "hyperv.h"
 #include "vmx.h"
 
+#define L2_HCALL_INPUT_GPA	0x100000000ull
+
 static int ud_count;
 
 static void guest_ud_handler(struct ex_regs *regs)
@@ -60,6 +62,8 @@ void l2_guest_code(void)
 	rdmsr_from_l2(MSR_GS_BASE); /* intercepted */
 
 	/* L2 TLB flush tests */
+	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+			 L2_HCALL_INPUT_GPA, 0);
 	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE | HV_HYPERCALL_FAST_BIT, 0x0,
 			 HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES | HV_FLUSH_ALL_PROCESSORS);
 	rdmsr_from_l2(MSR_FS_BASE);
@@ -228,7 +232,7 @@ static struct kvm_vcpu *save_restore_vm(struct kvm_vm *vm,
 int main(int argc, char *argv[])
 {
 	gva_t vmx_pages_gva = 0, hv_pages_gva = 0;
-	gva_t hcall_page;
+	gva_t hcall_input_page, hcall_page;
 
 	struct kvm_vcpu *vcpu;
 	struct kvm_vm *vm;
@@ -239,17 +243,23 @@ int main(int argc, char *argv[])
 	TEST_REQUIRE(kvm_has_cap(KVM_CAP_NESTED_STATE));
 	TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_ENLIGHTENED_VMCS));
 	TEST_REQUIRE(kvm_hv_cpu_has(HV_X64_NESTED_DIRECT_FLUSH));
+	TEST_REQUIRE(kvm_cpu_has_ept());
 
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	vm_enable_ept(vm);
 
 	hcall_page = vm_alloc_pages(vm, 1);
 	memset(addr_gva2hva(vm, hcall_page), 0x0,  getpagesize());
+	hcall_input_page = vm_alloc_page(vm);
+	memset(addr_gva2hva(vm, hcall_input_page), 0, getpagesize());
 
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_enable_evmcs(vcpu);
 
 	vcpu_alloc_vmx(vm, &vmx_pages_gva);
 	vcpu_alloc_hyperv_test_pages(vm, &hv_pages_gva);
+	tdp_identity_map_default_memslots(vm);
+	tdp_map(vm, L2_HCALL_INPUT_GPA, addr_gva2gpa(vm, hcall_input_page), PAGE_SIZE);
 	vcpu_args_set(vcpu, 3, vmx_pages_gva, hv_pages_gva, addr_gva2gpa(vm, hcall_page));
 	vcpu_set_msr(vcpu, HV_X64_MSR_VP_INDEX, vcpu->id);
 
tools/testing/selftests/kvm/x86/hyperv_svm_test.c +12 -1
index 1f74b0fa9b83..18f0ad6debd8 100644
@@ -18,6 +18,8 @@
 #include "svm_util.h"
 #include "hyperv.h"
 
+#define L2_HCALL_INPUT_GPA	0x100000000ull
+
 /* Exit to L1 from L2 with RDMSR instruction */
 static inline void rdmsr_from_l2(u32 msr)
 {
@@ -45,6 +47,8 @@ void l2_guest_code(void)
 	GUEST_SYNC(5);
 
 	/* L2 TLB flush tests */
+	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE,
+			 L2_HCALL_INPUT_GPA, 0);
 	hyperv_hypercall(HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE |
 			 HV_HYPERCALL_FAST_BIT, 0x0,
 			 HV_FLUSH_ALL_VIRTUAL_ADDRESS_SPACES |
@@ -146,7 +150,7 @@ static void __attribute__((__flatten__)) guest_code(struct svm_test_data *svm,
 int main(int argc, char *argv[])
 {
 	gva_t nested_gva = 0, hv_pages_gva = 0;
-	gva_t hcall_page;
+	gva_t hcall_input_page, hcall_page;
 	struct kvm_vcpu *vcpu;
 	struct kvm_vm *vm;
 	struct ucall uc;
@@ -154,15 +158,22 @@ int main(int argc, char *argv[])
 
 	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM));
 	TEST_REQUIRE(kvm_hv_cpu_has(HV_X64_NESTED_DIRECT_FLUSH));
+	TEST_REQUIRE(kvm_cpu_has_npt());
 
 	/* Create VM */
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	vm_enable_npt(vm);
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_alloc_svm(vm, &nested_gva);
 	vcpu_alloc_hyperv_test_pages(vm, &hv_pages_gva);
 
 	hcall_page = vm_alloc_pages(vm, 1);
 	memset(addr_gva2hva(vm, hcall_page), 0x0,  getpagesize());
+	hcall_input_page = vm_alloc_page(vm);
+	memset(addr_gva2hva(vm, hcall_input_page), 0, getpagesize());
+
+	tdp_identity_map_default_memslots(vm);
+	tdp_map(vm, L2_HCALL_INPUT_GPA, addr_gva2gpa(vm, hcall_input_page), PAGE_SIZE);
 
 	vcpu_args_set(vcpu, 3, nested_gva, hv_pages_gva, addr_gva2gpa(vm, hcall_page));
 	vcpu_set_msr(vcpu, HV_X64_MSR_VP_INDEX, vcpu->id);
Signature · 1 line
2.52.0
author maintainer
#5 lore raw

PATCH 5/31 KVM: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING

In reply to Paolo Bonzini

KVM_HC_CLOCK_PAIRING writes to a guest physical addresses. Treat it as an L2 address when running nested, which is consistent with the Hyper-V TLB flush hypercalls for example.

This is technically backwards incompatible, but it should be extremely unlikely to happen in the wild and thus we can treat it as a bugfix, I think.

---
 arch/x86/kvm/x86.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)
arch/x86/kvm/x86.c +14 -6
index 4b3681796c75..0fbe2d4e685f 100644
@@ -7195,8 +7195,8 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 {
 	struct kvm_clock_pairing clock_pairing;
 	struct timespec64 ts;
+	size_t offset = 0;
 	u64 cycle;
-	int ret;
 
 	if (clock_type != KVM_CLOCK_PAIRING_WALLCLOCK)
 		return -KVM_EOPNOTSUPP;
@@ -7217,12 +7217,20 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 	clock_pairing.flags = 0;
 	memset(&clock_pairing.pad, 0, sizeof(clock_pairing.pad));
 
-	ret = 0;
-	if (kvm_write_guest(vcpu->kvm, paddr, &clock_pairing,
-			    sizeof(struct kvm_clock_pairing)))
-		ret = -KVM_EFAULT;
+	while (offset < sizeof(clock_pairing)) {
+		gpa_t gpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, paddr + offset,
+					      PFERR_WRITE_MASK | PFERR_GUEST_FINAL_MASK, NULL, 0);
+		size_t len = min_t(size_t, sizeof(clock_pairing) - offset,
+				   PAGE_SIZE - offset_in_page(gpa));
 
-	return ret;
+		if (gpa == INVALID_GPA)
+			return -KVM_EFAULT;
+		if (kvm_write_guest(vcpu->kvm, gpa, (u8 *)&clock_pairing + offset, len))
+			return -KVM_EFAULT;
+
+		offset += len;
+	}
+	return 0;
 }
 #endif
 
Signature · 1 line
2.52.0
author maintainer
#6 lore raw

PATCH 6/31 KVM: x86: Introduce memory fault on invalid hypercalls reads/writes

In reply to Paolo Bonzini

Allow userspace to perform arbitrary actions when a hypercall refers to an invalid address, by exiting with KVM_EXIT_MEMORY_FAULT. This will for example allow userspace to perform a VTL call.

---
 Documentation/virt/kvm/api.rst  |  19 ++++
 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/kvm/hyperv.c           | 157 +++++++++++++++++++++++++-------
 arch/x86/kvm/x86.c              |  34 +++++--
 include/uapi/linux/kvm.h        |   1 +
 5 files changed, 173 insertions(+), 39 deletions(-)
Documentation/virt/kvm/api.rst +19 -0
index 37028ce019e3..abc2ff1f8c84 100644
@@ -8984,6 +8984,25 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key
 interpretation are disabled. If cmma has already been enabled or the
 hpage_2g module parameter is not set to 1, -EINVAL is returned.
 
+7.48 KVM_CAP_HCALL_FAULT_EXIT
+------------------------------------
+
+:Architectures: x86
+:Parameters: args[0] is 0 to disable, 1 to enable
+
+When enabled, KVM checks the memory that is read or written by
+hypercalls (including slow Hyper-V hypercalls and KVM_HC_CLOCK_PAIRING).
+An inaccessible input page causes a KVM_EXIT_MEMORY_FAULT with
+KVM_MEMORY_EXIT_FLAG_READ.  An inaccessible or read-only output page causes
+a KVM_EXIT_MEMORY_FAULT with KVM_MEMORY_EXIT_FLAG_WRITE.  The reported
+range identifies the page containing the parameter GPA.
+
+Hypercall parameters that are unused by the selected hypercall are not checked.
+
+For Hyper-V, unknown hypercalls are passed to userspace without checking their
+parameter pages.  The fault GPA is in the physical address space of the
+VM managed by userspace, after nested GPA translation.
+
 8. Other capabilities.
 ======================
 
arch/x86/include/asm/kvm_host.h +1 -0
index 683bb8bf43a9..c08781bb0327 100644
@@ -1169,6 +1169,7 @@ struct kvm_arch {
 	bool has_protected_eoi;
 	bool has_protected_pmu;
 	bool pre_fault_allowed;
+	bool hcall_fault_exit;
 	struct hlist_head *mmu_page_hash;
 	struct list_head active_mmu_pages;
 	struct kvm_possible_nx_huge_pages possible_nx_huge_pages[KVM_NR_MMU_TYPES];
arch/x86/kvm/hyperv.c +125 -32

256 lines not on this page · raw message

arch/x86/kvm/x86.c +27 -7
index 0fbe2d4e685f..3338d85c721f 100644
@@ -2242,6 +2242,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_HYPERV_ENFORCE_CPUID:
 	case KVM_CAP_SYS_HYPERV_CPUID:
 #endif
+	case KVM_CAP_HCALL_FAULT_EXIT:
 	case KVM_CAP_PCI_SEGMENT:
 	case KVM_CAP_DEBUGREGS:
 	case KVM_CAP_X86_ROBUST_SINGLESTEP:
@@ -4210,6 +4211,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		mutex_unlock(&kvm->lock);
 		break;
 	}
+	case KVM_CAP_HCALL_FAULT_EXIT:
+		kvm->arch.hcall_fault_exit = cap->args[0];
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -7191,25 +7196,28 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_vendor_exit);
 
 #ifdef CONFIG_X86_64
 static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
-			        unsigned long clock_type)
+			        unsigned long clock_type, unsigned long *ret)
 {
 	struct kvm_clock_pairing clock_pairing;
 	struct timespec64 ts;
 	size_t offset = 0;
+	bool writable;
+	unsigned long hva;
 	u64 cycle;
 
+	*ret = -KVM_EOPNOTSUPP;
 	if (clock_type != KVM_CLOCK_PAIRING_WALLCLOCK)
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	/*
 	 * When tsc is in permanent catchup mode guests won't be able to use
 	 * pvclock_read_retry loop to get consistent view of pvclock
 	 */
 	if (vcpu->arch.tsc_always_catchup)
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	if (!kvm_get_walltime_and_clockread(&ts, &cycle))
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	clock_pairing.sec = ts.tv_sec;
 	clock_pairing.nsec = ts.tv_nsec;
@@ -7217,6 +7225,7 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 	clock_pairing.flags = 0;
 	memset(&clock_pairing.pad, 0, sizeof(clock_pairing.pad));
 
+	*ret = -KVM_EFAULT;
 	while (offset < sizeof(clock_pairing)) {
 		gpa_t gpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, paddr + offset,
 					      PFERR_WRITE_MASK | PFERR_GUEST_FINAL_MASK, NULL, 0);
@@ -7224,12 +7233,22 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 				   PAGE_SIZE - offset_in_page(gpa));
 
 		if (gpa == INVALID_GPA)
-			return -KVM_EFAULT;
+			return 0;
+		if (vcpu->kvm->arch.hcall_fault_exit) {
+			hva = kvm_vcpu_gfn_to_hva_prot(vcpu, gpa_to_gfn(gpa), &writable);
+			if (kvm_is_error_hva(hva) || !writable) {
+				kvm_prepare_memory_fault_exit(vcpu, gpa & PAGE_MASK, PAGE_SIZE,
+							      true, false, false);
+				return -EFAULT;
+			}
+		}
 		if (kvm_write_guest(vcpu->kvm, gpa, (u8 *)&clock_pairing + offset, len))
-			return -KVM_EFAULT;
+			return 0;
 
 		offset += len;
 	}
+
+	*ret = 0;
 	return 0;
 }
 #endif
@@ -7388,7 +7407,8 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl,
 		break;
 #ifdef CONFIG_X86_64
 	case KVM_HC_CLOCK_PAIRING:
-		ret = kvm_pv_clock_pairing(vcpu, a0, a1);
+		if (kvm_pv_clock_pairing(vcpu, a0, a1, &ret))
+			return -EFAULT;
 		break;
 #endif
 	case KVM_HC_SEND_IPI:
include/uapi/linux/kvm.h +1 -0
index 15a3090f067e..c23f1ea62eaf 100644
@@ -1002,6 +1002,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_HPAGE_2G 249
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
+#define KVM_CAP_HCALL_FAULT_EXIT 252
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
Signature · 1 line
2.52.0
#7 lore raw

Re: [PATCH 06/31] KVM: x86: Introduce memory fault on invalid hypercalls reads/writes

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] kvm: KVM_CAP_HCALL_FAULT_EXIT can be toggled dynamically at runtime without locks --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] kvm: KVM_CAP_HCALL_FAULT_EXIT can be toggled dynamically at runtime without locks --

commit 2924fdad4ef4a10a3e4ce84aa7cf7e1887f8928d
Author: Paolo Bonzini <[email protected]>

KVM: x86: Introduce memory fault on invalid hypercalls reads/writes

This commit introduces the KVM_CAP_HCALL_FAULT_EXIT capability to check memory read or written by hypercalls. When enabled, an inaccessible input or output page causes an exit with KVM_EXIT_MEMORY_FAULT. This allows userspace to perform arbitrary actions when a hypercall refers to an invalid address.

Quoted 4 lines
arch/x86/kvm/x86.c +0 -0
index 1b8f24577dbf5..9206593878ee9 100644

[ ... ]

arch/x86/kvm/x86.c int kvm_vm_ioctl_enable_cap(struct kvm *kvm, · line 4215

Quoted 6 lines
arch/x86/kvm/x86.c +2 -0
@@ -4210,6 +4211,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		mutex_unlock(&kvm->lock);
 		break;
 	}
+	case KVM_CAP_HCALL_FAULT_EXIT:
+		kvm->arch.hcall_fault_exit = cap->args[0];

[Severity: Medium]
Is it safe to toggle this capability dynamically without synchronization?

According to the KVM review guidelines, new state flags and capabilities that modify behavior should default to immutable to prevent unanticipated state machine transitions. Modifying this flag without acquiring the VM lock or verifying if vCPUs have already been created allows userspace to dynamically flip the capability while vCPUs are running.

Does this also introduce a data race with concurrent readers in the hypercall handlers, since the capability flag is written without atomic annotations or locks?

Quoted 5 lines
arch/x86/kvm/x86.c +2 -0
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
maintainer
#8 lore raw

Re: [PATCH 06/31] KVM: x86: Introduce memory fault on invalid hypercalls reads/writes

In reply to Paolo Bonzini
Quoted 31 lines · Paolo Bonzini

Allow userspace to perform arbitrary actions when a hypercall refers to an invalid address, by exiting with KVM_EXIT_MEMORY_FAULT. This will for example allow userspace to perform a VTL call.

---
 Documentation/virt/kvm/api.rst  |  19 ++++
 arch/x86/include/asm/kvm_host.h |   1 +
 arch/x86/kvm/hyperv.c           | 157 +++++++++++++++++++++++++-------
 arch/x86/kvm/x86.c              |  34 +++++--
 include/uapi/linux/kvm.h        |   1 +
 5 files changed, 173 insertions(+), 39 deletions(-)
Documentation/virt/kvm/api.rst +8 -0
index 37028ce019e3..abc2ff1f8c84 100644
@@ -8984,6 +8984,25 @@ enabled, cmma can't be enabled anymore and pfmfi and the storage key
 interpretation are disabled. If cmma has already been enabled or the
 hpage_2g module parameter is not set to 1, -EINVAL is returned.
 
+7.48 KVM_CAP_HCALL_FAULT_EXIT
+------------------------------------
+
+:Architectures: x86
+:Parameters: args[0] is 0 to disable, 1 to enable
+
+When enabled, KVM checks the memory that is read or written by
+hypercalls (including slow Hyper-V hypercalls and KVM_HC_CLOCK_PAIRING).

Documentation/virt/kvm/api.rst enabled, cmma can't be enabled anymore and pfmfi and the storage key · line 9004

Quoted 10 lines
Documentation/virt/kvm/api.rst +10 -0
+An inaccessible input page causes a KVM_EXIT_MEMORY_FAULT with
+KVM_MEMORY_EXIT_FLAG_READ.  An inaccessible or read-only output page causes
+a KVM_EXIT_MEMORY_FAULT with KVM_MEMORY_EXIT_FLAG_WRITE.  The reported
+range identifies the page containing the parameter GPA.
+
+Hypercall parameters that are unused by the selected hypercall are not checked.
+
+For Hyper-V, unknown hypercalls are passed to userspace without checking their
+parameter pages.  The fault GPA is in the physical address space of the
+VM managed by userspace, after nested GPA translation.

I'm sure you thought it through but since this is going to be a hard-to-change API let me ask: would it make any sense to try to keep Hyper-V world separate here, i.e. have a dedicated KVM_CAP_HYPERV_HCALL_FAULT_EXIT so VMMs can implement and enable/disable them separately? (I don't personally see a big benefit from VMM's PoV, but still).

Quoted 401 lines
Documentation/virt/kvm/api.rst +1 -0
+
 8. Other capabilities.
 ======================
 
arch/x86/include/asm/kvm_host.h +1 -0
index 683bb8bf43a9..c08781bb0327 100644
@@ -1169,6 +1169,7 @@ struct kvm_arch {
 	bool has_protected_eoi;
 	bool has_protected_pmu;
 	bool pre_fault_allowed;
+	bool hcall_fault_exit;
 	struct hlist_head *mmu_page_hash;
 	struct list_head active_mmu_pages;
 	struct kvm_possible_nx_huge_pages possible_nx_huge_pages[KVM_NR_MMU_TYPES];
arch/x86/kvm/hyperv.c +125 -32
index d2921d443fde..e3a8e8236230 100644
@@ -2033,7 +2033,68 @@ int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu)
 	return -ENOSPC;
 }
 
-static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+static int kvm_hv_hypercall_check_gpa(struct kvm_vcpu *vcpu, gpa_t gpa,
+				      bool write)
+{
+	bool writable = true;
+	gfn_t gfn = gpa_to_gfn(gpa);
+	unsigned long addr;
+
+	addr = kvm_vcpu_gfn_to_hva_prot(vcpu, gfn, &writable);
+	if (!kvm_is_error_hva(addr) && (!write || writable))
+		return 0;
+
+	kvm_prepare_memory_fault_exit(vcpu, gfn_to_gpa(gfn), PAGE_SIZE,
+				      write, false, false);
+	return -EFAULT;
+}
+
+static unsigned int kvm_hv_hypercall_mem_access(u16 code)
+{
+	switch (code) {
+	case HVCALL_SIGNAL_EVENT:
+	case HVCALL_POST_MESSAGE:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST_EX:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE:
+	case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE_EX:
+	case HVCALL_SEND_IPI:
+	case HVCALL_SEND_IPI_EX:
+		return KVM_MEMORY_EXIT_FLAG_READ;
+	case HVCALL_POST_DEBUG_DATA:
+	case HVCALL_RETRIEVE_DEBUG_DATA:
+		return KVM_MEMORY_EXIT_FLAG_READ | KVM_MEMORY_EXIT_FLAG_WRITE;
+	case HVCALL_RESET_DEBUG_SESSION:
+	case HV_EXT_CALL_QUERY_CAPABILITIES:
+		return KVM_MEMORY_EXIT_FLAG_WRITE;
+	}
+
+	return 0;
+}
+
+static int kvm_hv_hypercall_check_params(struct kvm_vcpu *vcpu,
+					 struct kvm_hv_hcall *hc)
+{
+	unsigned access;
+	int r;
+
+	if (hc->fast || !vcpu->kvm->arch.hcall_fault_exit)
+		return 0;
+
+	access = kvm_hv_hypercall_mem_access(hc->code);
+	if (access & KVM_MEMORY_EXIT_FLAG_READ) {
+		r = kvm_hv_hypercall_check_gpa(vcpu, hc->ingpa, false);
+		if (r)
+			return r;
+	}
+
+	if (access & KVM_MEMORY_EXIT_FLAG_WRITE)
+		return kvm_hv_hypercall_check_gpa(vcpu, hc->outgpa, true);
+
+	return 0;
+}
+
+static s64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	unsigned long *vcpu_mask = hv_vcpu->vcpu_mask;
@@ -2053,6 +2114,19 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	struct kvm_vcpu *v;
 	unsigned long i;
 	bool all_cpus;
+	int r;
+
+	/* Slow direct hypercalls from L2 provide a nested GPA. */
+	if (!hc->fast) {
+		hc->ingpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, hc->ingpa,
+					      PFERR_GUEST_FINAL_MASK, NULL, 0);
+		if (unlikely(hc->ingpa == INVALID_GPA))
+			return HV_STATUS_INVALID_HYPERCALL_INPUT;
+	}
+
+	r = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (r)
+		return r;
 
 	/*
 	 * The Hyper-V TLFS doesn't allow more than HV_MAX_SPARSE_VCPU_BANKS
@@ -2061,20 +2135,6 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	 */
 	BUILD_BUG_ON(KVM_HV_MAX_SPARSE_VCPU_SET_BITS > HV_MAX_SPARSE_VCPU_BANKS);
 
-	/*
-	 * 'Slow' hypercall's first parameter is the address in guest's memory
-	 * where hypercall parameters are placed. This is either a GPA or a
-	 * nested GPA when KVM is handling the call from L2 ('direct' TLB
-	 * flush).  Translate the address here so the memory can be uniformly
-	 * read with kvm_read_guest().
-	 */
-	if (!hc->fast) {
-		hc->ingpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, hc->ingpa,
-					      PFERR_GUEST_FINAL_MASK, NULL, 0);
-		if (unlikely(hc->ingpa == INVALID_GPA))
-			return HV_STATUS_INVALID_HYPERCALL_INPUT;
-	}
-
 	if (hc->code == HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST ||
 	    hc->code == HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE) {
 		if (hc->fast) {
@@ -2242,7 +2302,7 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, u32 vector,
 	}
 }
 
-static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+static int kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	u64 *sparse_banks = hv_vcpu->sparse_banks;
@@ -2252,6 +2312,11 @@ static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 	u64 valid_bank_mask;
 	u32 vector;
 	bool all_cpus;
+	int r;
+
+	r = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (r)
+		return r;
 
 	if (!lapic_in_kernel(vcpu))
 		return HV_STATUS_INVALID_HYPERCALL_INPUT;
@@ -2432,11 +2497,16 @@ static int kvm_hv_hypercall_complete_userspace(struct kvm_vcpu *vcpu)
 	return kvm_hv_hypercall_complete(vcpu, vcpu->run->hyperv.u.hcall.result);
 }
 
-static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
+static int kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc)
 {
 	struct kvm_hv *hv = to_kvm_hv(vcpu->kvm);
 	struct eventfd_ctx *eventfd;
 	u64 conn_id;
+	int ret;
+
+	ret = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (ret)
+		return ret;
 
 	if (unlikely(!hc->fast)) {
 		int ret;
@@ -2550,11 +2620,30 @@ static bool hv_check_hypercall_access(struct kvm_vcpu_hv *hv_vcpu, u16 code)
 	return true;
 }
 
+static int kvm_hv_hypercall_userspace_exit(struct kvm_vcpu *vcpu,
+					   struct kvm_hv_hcall *hc)
+{
+	int r;
+
+	r = kvm_hv_hypercall_check_params(vcpu, hc);
+	if (r)
+		return r;
+
+	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
+	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
+	vcpu->run->hyperv.u.hcall.input = hc->param;
+	vcpu->run->hyperv.u.hcall.params[0] = hc->ingpa;
+	vcpu->run->hyperv.u.hcall.params[1] = hc->outgpa;
+	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
+	return 0;
+}
+
 int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu);
 	struct kvm_hv_hcall hc;
 	u64 ret = HV_STATUS_SUCCESS;
+	s64 r;
 
 	/*
 	 * hypercall generates UD from non zero cpl and real mode
@@ -2622,7 +2711,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hvcall_signal_event(vcpu, &hc);
+		r = kvm_hvcall_signal_event(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		if (ret != HV_STATUS_INVALID_PORT_ID)
 			break;
 		fallthrough;	/* maybe userspace knows this conn_id */
@@ -2632,7 +2724,7 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		goto hypercall_userspace_exit;
+		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	case HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST:
 		if (unlikely(hc.var_cnt)) {
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
@@ -2644,7 +2736,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hv_flush_tlb(vcpu, &hc);
+		r = kvm_hv_flush_tlb(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		break;
 	case HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE:
 		if (unlikely(hc.var_cnt)) {
@@ -2657,7 +2752,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hv_flush_tlb(vcpu, &hc);
+		r = kvm_hv_flush_tlb(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		break;
 	case HVCALL_SEND_IPI:
 		if (unlikely(hc.var_cnt)) {
@@ -2670,7 +2768,10 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_INVALID_HYPERCALL_INPUT;
 			break;
 		}
-		ret = kvm_hv_send_ipi(vcpu, &hc);
+		r = kvm_hv_send_ipi(vcpu, &hc);
+		if (r < 0)
+			return r;
+		ret = r;
 		break;
 	case HVCALL_POST_DEBUG_DATA:
 	case HVCALL_RETRIEVE_DEBUG_DATA:
@@ -2691,14 +2792,14 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 			ret = HV_STATUS_OPERATION_DENIED;
 			break;
 		}
-		goto hypercall_userspace_exit;
+		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	}
 	case HV_EXT_CALL_QUERY_CAPABILITIES ... HV_EXT_CALL_MAX:
 		if (unlikely(hc.fast)) {
 			ret = HV_STATUS_INVALID_PARAMETER;
 			break;
 		}
-		goto hypercall_userspace_exit;
+		return kvm_hv_hypercall_userspace_exit(vcpu, &hc);
 	default:
 		ret = HV_STATUS_INVALID_HYPERCALL_CODE;
 		break;
@@ -2707,14 +2808,6 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu)
 hypercall_complete:
 	return kvm_hv_hypercall_complete(vcpu, ret);
 
-hypercall_userspace_exit:
-	vcpu->run->exit_reason = KVM_EXIT_HYPERV;
-	vcpu->run->hyperv.type = KVM_EXIT_HYPERV_HCALL;
-	vcpu->run->hyperv.u.hcall.input = hc.param;
-	vcpu->run->hyperv.u.hcall.params[0] = hc.ingpa;
-	vcpu->run->hyperv.u.hcall.params[1] = hc.outgpa;
-	vcpu->arch.complete_userspace_io = kvm_hv_hypercall_complete_userspace;
-	return 0;
 }
 
 void kvm_hv_init_vm(struct kvm *kvm)
arch/x86/kvm/x86.c +27 -7
index 0fbe2d4e685f..3338d85c721f 100644
@@ -2242,6 +2242,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_HYPERV_ENFORCE_CPUID:
 	case KVM_CAP_SYS_HYPERV_CPUID:
 #endif
+	case KVM_CAP_HCALL_FAULT_EXIT:
 	case KVM_CAP_PCI_SEGMENT:
 	case KVM_CAP_DEBUGREGS:
 	case KVM_CAP_X86_ROBUST_SINGLESTEP:
@@ -4210,6 +4211,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		mutex_unlock(&kvm->lock);
 		break;
 	}
+	case KVM_CAP_HCALL_FAULT_EXIT:
+		kvm->arch.hcall_fault_exit = cap->args[0];
+		r = 0;
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -7191,25 +7196,28 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_vendor_exit);
 
 #ifdef CONFIG_X86_64
 static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
-			        unsigned long clock_type)
+			        unsigned long clock_type, unsigned long *ret)
 {
 	struct kvm_clock_pairing clock_pairing;
 	struct timespec64 ts;
 	size_t offset = 0;
+	bool writable;
+	unsigned long hva;
 	u64 cycle;
 
+	*ret = -KVM_EOPNOTSUPP;
 	if (clock_type != KVM_CLOCK_PAIRING_WALLCLOCK)
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	/*
 	 * When tsc is in permanent catchup mode guests won't be able to use
 	 * pvclock_read_retry loop to get consistent view of pvclock
 	 */
 	if (vcpu->arch.tsc_always_catchup)
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	if (!kvm_get_walltime_and_clockread(&ts, &cycle))
-		return -KVM_EOPNOTSUPP;
+		return 0;
 
 	clock_pairing.sec = ts.tv_sec;
 	clock_pairing.nsec = ts.tv_nsec;
@@ -7217,6 +7225,7 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 	clock_pairing.flags = 0;
 	memset(&clock_pairing.pad, 0, sizeof(clock_pairing.pad));
 
+	*ret = -KVM_EFAULT;
 	while (offset < sizeof(clock_pairing)) {
 		gpa_t gpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, paddr + offset,
 					      PFERR_WRITE_MASK | PFERR_GUEST_FINAL_MASK, NULL, 0);
@@ -7224,12 +7233,22 @@ static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr,
 				   PAGE_SIZE - offset_in_page(gpa));
 
 		if (gpa == INVALID_GPA)
-			return -KVM_EFAULT;
+			return 0;
+		if (vcpu->kvm->arch.hcall_fault_exit) {
+			hva = kvm_vcpu_gfn_to_hva_prot(vcpu, gpa_to_gfn(gpa), &writable);
+			if (kvm_is_error_hva(hva) || !writable) {
+				kvm_prepare_memory_fault_exit(vcpu, gpa & PAGE_MASK, PAGE_SIZE,
+							      true, false, false);
+				return -EFAULT;
+			}
+		}
 		if (kvm_write_guest(vcpu->kvm, gpa, (u8 *)&clock_pairing + offset, len))
-			return -KVM_EFAULT;
+			return 0;
 
 		offset += len;
 	}
+
+	*ret = 0;
 	return 0;
 }
 #endif
@@ -7388,7 +7407,8 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl,
 		break;
 #ifdef CONFIG_X86_64
 	case KVM_HC_CLOCK_PAIRING:
-		ret = kvm_pv_clock_pairing(vcpu, a0, a1);
+		if (kvm_pv_clock_pairing(vcpu, a0, a1, &ret))
+			return -EFAULT;
 		break;
 #endif
 	case KVM_HC_SEND_IPI:
include/uapi/linux/kvm.h +1 -0
index 15a3090f067e..c23f1ea62eaf 100644
@@ -1002,6 +1002,7 @@ struct kvm_enable_cap {
 #define KVM_CAP_S390_HPAGE_2G 249
 #define KVM_CAP_PPC_COMPAT_CAPS 250
 #define KVM_CAP_ARM_PMU_V3_STRICT 251
+#define KVM_CAP_HCALL_FAULT_EXIT 252
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
Signature · 1 line
Vitaly
author maintainer
#9 lore raw

PATCH 7/31 KVM: selftests: test hypercall memory fault exits

In reply to Paolo Bonzini

Cover various scenarios where hypercalls are invoked with invalid or read-only GPAs, and check that they exit to userspace.

---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 tools/testing/selftests/kvm/x86/hcall_fault.c | 246 ++++++++++++++++++
 .../testing/selftests/kvm/x86/hyperv_evmcs.c  |   4 +
 .../selftests/kvm/x86/hyperv_svm_test.c       |   2 +
 4 files changed, 253 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/hcall_fault.c
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 96bab7002d39..6e2bbf743eaa 100644
@@ -79,6 +79,7 @@ TEST_GEN_PROGS_x86 += x86/evmcs_smm_controls_test
 TEST_GEN_PROGS_x86 += x86/exit_on_emulation_failure_test
 TEST_GEN_PROGS_x86 += x86/fastops_test
 TEST_GEN_PROGS_x86 += x86/fix_hypercall_test
+TEST_GEN_PROGS_x86 += x86/hcall_fault
 TEST_GEN_PROGS_x86 += x86/hwcr_msr_test
 TEST_GEN_PROGS_x86 += x86/hyperv_clock
 TEST_GEN_PROGS_x86 += x86/hyperv_cpuid
tools/testing/selftests/kvm/x86/hcall_fault.c new file +246 -0

246 lines not on this page · raw message

tools/testing/selftests/kvm/x86/hyperv_evmcs.c +4 -0
index 29baae74ad3a..59d200ccb408 100644
@@ -216,6 +216,8 @@ static struct kvm_vcpu *save_restore_vm(struct kvm_vm *vm,
 
 	/* Restore state in a new VM.  */
 	vcpu = vm_recreate_with_one_vcpu(vm);
+	if (kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT))
+		vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_enable_evmcs(vcpu);
 	vcpu_load_state(vcpu, state);
@@ -246,6 +248,8 @@ int main(int argc, char *argv[])
 	TEST_REQUIRE(kvm_cpu_has_ept());
 
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	if (kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT))
+		vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
 	vm_enable_ept(vm);
 
 	hcall_page = vm_alloc_pages(vm, 1);
tools/testing/selftests/kvm/x86/hyperv_svm_test.c +2 -0
index 18f0ad6debd8..ecad242e6623 100644
@@ -162,6 +162,8 @@ int main(int argc, char *argv[])
 
 	/* Create VM */
 	vm = vm_create_with_one_vcpu(&vcpu, guest_code);
+	if (kvm_has_cap(KVM_CAP_HCALL_FAULT_EXIT))
+		vm_enable_cap(vm, KVM_CAP_HCALL_FAULT_EXIT, 1);
 	vm_enable_npt(vm);
 	vcpu_set_hv_cpuid(vcpu);
 	vcpu_alloc_svm(vm, &nested_gva);
Signature · 1 line
2.52.0
author maintainer
#10 lore raw

PATCH 8/31 KVM: x86/mmu: intersect writability from __kvm_faultin_pfn with fault->map_writable

In reply to Paolo Bonzini

fault->map_writable is currently a pure output of __kvm_faultin_pfn(), which is the only thing that restricts it. This will no longer hold once memory protections derived from memory attributes are applied: those compute their own access permissions that combine with those from __kvm_faultin_pfn().

Applying them *before* faulting in the pfn lets a fault that violates the attributes exit to userspace without the cost of gup and/or an async #PF; but it means that permissions will then be restricted in two independent steps, first by memory attributes and then by __kvm_faultin_pfn(). Switch fault->map_writable to that model by letting kvm_mmu_faultin_pfn() only clear bits rather than assign them.

No functional change intended: nothing writes fault->map_writable between the initializer and __kvm_mmu_faultin_pfn() yet, so the AND is equivalent to the assignment it replaces.

---
 arch/x86/kvm/mmu/mmu.c | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
arch/x86/kvm/mmu/mmu.c +8 -4
index 064ecc33b926..5996468b7120 100644
@@ -4634,7 +4634,7 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 		return r;
 	}
 
-	fault->map_writable = !(fault->slot->flags & KVM_MEM_READONLY);
+	fault->map_writable &= !(fault->slot->flags & KVM_MEM_READONLY);
 	fault->max_level = kvm_max_level_for_order(max_order);
 
 	return RET_PF_CONTINUE;
@@ -4644,13 +4644,14 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 				 struct kvm_page_fault *fault)
 {
 	unsigned int foll = fault->write ? FOLL_WRITE : 0;
+	bool writable;
 
 	if (fault->is_private || kvm_memslot_is_gmem_only(fault->slot))
 		return kvm_mmu_faultin_pfn_gmem(vcpu, fault);
 
 	foll |= FOLL_NOWAIT;
 	fault->pfn = __kvm_faultin_pfn(fault->slot, fault->gfn, foll,
-				       &fault->map_writable, &fault->refcounted_page);
+				       &writable, &fault->refcounted_page);
 
 	/*
 	 * If resolving the page failed because I/O is needed to fault-in the
@@ -4659,7 +4660,7 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	 * other failures are terminal, i.e. retrying won't help.
 	 */
 	if (fault->pfn != KVM_PFN_ERR_NEEDS_IO)
-		return RET_PF_CONTINUE;
+		goto out_pf_continue;
 
 	if (!fault->prefetch && kvm_can_do_async_pf(vcpu)) {
 		trace_kvm_try_async_get_page(fault->addr, fault->gfn);
@@ -4680,8 +4681,10 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	foll |= FOLL_INTERRUPTIBLE;
 	foll &= ~FOLL_NOWAIT;
 	fault->pfn = __kvm_faultin_pfn(fault->slot, fault->gfn, foll,
-				       &fault->map_writable, &fault->refcounted_page);
+				       &writable, &fault->refcounted_page);
 
+out_pf_continue:
+	fault->map_writable &= writable;
 	return RET_PF_CONTINUE;
 }
 
@@ -5000,6 +5003,7 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.is_private = err & PFERR_PRIVATE_ACCESS,
 
 		.pfn = KVM_PFN_ERR_FAULT,
+		.map_writable = true,
 	};
 	int r;
 
Signature · 1 line
2.52.0
author maintainer
#11 lore raw

PATCH 9/31 KVM: x86/mmu: Extend map_writable to a full ACC_* mask

In reply to Paolo Bonzini

Support for memory protection attributes opens the door to installing non-executable mappings. Instead of introducing yet another member in struct kvm_page_fault and another argument to make_spte(), make the existing member map_writable a mask of ACC_* bits. This also avoids the need for make_spte() to map a single bool to either the NX bit or the XS/XU bits together.

Unlike for mappings that are not writable because the fault did not request write premission, it is not not necessary to track executability for these SPTEs; the gfn is always available and it will be possible to access the attributes directly in FNAME(sync_spte).

---
 arch/x86/kvm/mmu/mmu.c          | 21 ++++++++++++---------
 arch/x86/kvm/mmu/mmu_internal.h |  2 +-
 arch/x86/kvm/mmu/paging_tmpl.h  |  8 +++++---
 arch/x86/kvm/mmu/spte.c         | 12 ++++++------
 arch/x86/kvm/mmu/spte.h         |  2 +-
 arch/x86/kvm/mmu/tdp_mmu.c      |  2 +-
 6 files changed, 26 insertions(+), 21 deletions(-)
arch/x86/kvm/mmu/mmu.c +12 -9
index 5996468b7120..b72ccbee0d86 100644
@@ -3105,7 +3105,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot,
 	u64 spte;
 
 	/* Prefetching always gets a writable pfn.  */
-	bool host_writable = !fault || fault->map_writable;
+	unsigned host_access = fault ? fault->host_access : ACC_ALL;
 	bool prefetch = !fault || fault->prefetch;
 	bool write_fault = fault && fault->write;
 
@@ -3142,7 +3142,7 @@ static int mmu_set_spte(struct kvm_vcpu *vcpu, struct kvm_memory_slot *slot,
 	}
 
 	wrprot = make_spte(vcpu, sp, slot, pte_access, gfn, pfn, *sptep, prefetch,
-			   false, host_writable, &spte);
+			   false, host_access, &spte);
 
 	if (*sptep == spte) {
 		ret = RET_PF_SPURIOUS;
@@ -3589,7 +3589,7 @@ static int kvm_handle_noslot_fault(struct kvm_vcpu *vcpu,
 
 	fault->slot = NULL;
 	fault->pfn = KVM_PFN_NOSLOT;
-	fault->map_writable = false;
+	fault->host_access = 0;
 
 	/*
 	 * If MMIO caching is disabled, emulate immediately without
@@ -4614,7 +4614,8 @@ static void kvm_mmu_finish_page_fault(struct kvm_vcpu *vcpu,
 				      struct kvm_page_fault *fault, int r)
 {
 	kvm_release_faultin_page(vcpu->kvm, fault->refcounted_page,
-				 r == RET_PF_RETRY, fault->map_writable);
+				 r == RET_PF_RETRY,
+				 !!(fault->host_access & ACC_WRITE_MASK));
 }
 
 static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
@@ -4634,9 +4635,10 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 		return r;
 	}
 
-	fault->map_writable &= !(fault->slot->flags & KVM_MEM_READONLY);
-	fault->max_level = kvm_max_level_for_order(max_order);
+	if (fault->slot->flags & KVM_MEM_READONLY)
+		fault->host_access &= ~ACC_WRITE_MASK;
 
+	fault->max_level = kvm_max_level_for_order(max_order);
 	return RET_PF_CONTINUE;
 }
 
@@ -4684,7 +4686,8 @@ static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 				       &writable, &fault->refcounted_page);
 
 out_pf_continue:
-	fault->map_writable &= writable;
+	if (!writable)
+		fault->host_access &= ~ACC_WRITE_MASK;
 	return RET_PF_CONTINUE;
 }
 
@@ -5003,7 +5006,7 @@ static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 		.is_private = err & PFERR_PRIVATE_ACCESS,
 
 		.pfn = KVM_PFN_ERR_FAULT,
-		.map_writable = true,
+		.host_access = ACC_ALL,
 	};
 	int r;
 
@@ -5197,7 +5200,7 @@ int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)
 		.gfn = gfn,
 		.slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn),
 		.pfn = pfn,
-		.map_writable = true,
+		.host_access = ACC_ALL,
 	};
 	struct kvm *kvm = vcpu->kvm;
 	int r;
arch/x86/kvm/mmu/mmu_internal.h +1 -1
index c29002c60126..00215b9f309f 100644
@@ -280,7 +280,7 @@ struct kvm_page_fault {
 	unsigned long mmu_seq;
 	kvm_pfn_t pfn;
 	struct page *refcounted_page;
-	bool map_writable;
+	u8 host_access;
 
 	/*
 	 * Indicates the guest is trying to write a gfn that contains one or
arch/x86/kvm/mmu/paging_tmpl.h +5 -3
index 27427e7f22fa..e6ec14165f40 100644
@@ -935,7 +935,7 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
  */
 static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int i)
 {
-	bool host_writable;
+	u8 host_access;
 	gpa_t first_pte_gpa;
 	u64 *sptep, spte;
 	struct kvm_memory_slot *slot;
@@ -992,11 +992,13 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int
 
 	sptep = &sp->spt[i];
 	spte = *sptep;
-	host_writable = spte & shadow_host_writable_mask;
+	host_access = ACC_ALL;
+	if (!(spte & shadow_host_writable_mask))
+		host_access &= ~ACC_WRITE_MASK;
 	slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 	make_spte(vcpu, sp, slot, pte_access, gfn,
 		  spte_to_pfn(spte), spte, true, true,
-		  host_writable, &spte);
+		  host_access, &spte);
 
 	/*
 	 * There is no need to mark the pfn dirty, as the new protections must
arch/x86/kvm/mmu/spte.c +6 -6
index 5fc27e9733b3..1434164fa372 100644
@@ -189,7 +189,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	       const struct kvm_memory_slot *slot,
 	       unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn,
 	       u64 old_spte, bool prefetch, bool synchronizing,
-	       bool host_writable, u64 *new_spte)
+	       unsigned int host_access, u64 *new_spte)
 {
 	int level = sp->role.level;
 	u64 spte = SPTE_MMU_PRESENT_MASK;
@@ -207,6 +207,11 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	if (!prefetch || synchronizing)
 		spte |= shadow_accessed_mask;
 
+	if (host_access & ACC_WRITE_MASK)
+		spte |= shadow_host_writable_mask;
+
+	pte_access &= host_access;
+
 	/*
 	 * For simplicity, enforce the NX huge page mitigation even if not
 	 * strictly necessary.  KVM could ignore the mitigation if paging is
@@ -246,11 +251,6 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	if (kvm_x86_ops.get_mt_mask)
 		spte |= kvm_x86_call(get_mt_mask)(vcpu, gfn,
 						  kvm_is_mmio_pfn(pfn, &is_host_mmio));
-	if (host_writable)
-		spte |= shadow_host_writable_mask;
-	else
-		pte_access &= ~ACC_WRITE_MASK;
-
 	if (shadow_me_value && !kvm_is_mmio_pfn(pfn, &is_host_mmio))
 		spte |= shadow_me_value;
 
arch/x86/kvm/mmu/spte.h +1 -1
index e730717824b3..589f3954633e 100644
@@ -563,7 +563,7 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
 	       const struct kvm_memory_slot *slot,
 	       unsigned int pte_access, gfn_t gfn, kvm_pfn_t pfn,
 	       u64 old_spte, bool prefetch, bool synchronizing,
-	       bool host_writable, u64 *new_spte);
+	       unsigned int host_access, u64 *new_spte);
 u64 make_small_spte(struct kvm *kvm, u64 huge_spte,
 		    union kvm_mmu_page_role role, int index);
 u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level);
arch/x86/kvm/mmu/tdp_mmu.c +1 -1
index 44dad106fad1..dce44b9ce73a 100644
@@ -1143,7 +1143,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu,
 	else
 		wrprot = make_spte(vcpu, sp, fault->slot, sp->role.access, iter->gfn,
 				   fault->pfn, iter->old_spte, fault->prefetch,
-				   false, fault->map_writable, &new_spte);
+				   false, fault->host_access, &new_spte);
 
 	if (new_spte == iter->old_spte)
 		ret = RET_PF_SPURIOUS;
Signature · 1 line
2.52.0
author maintainer
#12 lore raw

PATCH 10/31 KVM: x86/mmu: Init memslot hugepage information for non-private_mem VMs too

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

The list of supported memory attributes is about to grow, and they will be available on any VM type (as opposed to only ones targeted at confidential computing). As such, update the check in kvm_mmu_init_memslot_memory_attributes() to initialize huge page information if any kind of memory attributes is available for the VM.

---
 arch/x86/kvm/mmu/mmu.c   | 2 +-
 include/linux/kvm_host.h | 5 +++++
 virt/kvm/kvm_main.c      | 2 +-
 3 files changed, 7 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -1
index b72ccbee0d86..ba462c671f7d 100644
@@ -8275,7 +8275,7 @@ void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
 {
 	int level;
 
-	if (!kvm_arch_has_private_mem(kvm))
+	if (!kvm_supported_mem_attributes(kvm))
 		return;
 
 	for (level = PG_LEVEL_2M; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) {
include/linux/kvm_host.h +5 -0
index ce5757e85ae3..acffb2f79b3e 100644
@@ -2579,12 +2579,17 @@ bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 					struct kvm_gfn_range *range);
 bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 					 struct kvm_gfn_range *range);
+u64 kvm_supported_mem_attributes(struct kvm *kvm);
 
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return kvm_get_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 }
 #else
+static inline u64 kvm_supported_mem_attributes(struct kvm *kvm)
+{
+	return 0;
+}
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;
virt/kvm/kvm_main.c +1 -1
index 65eb26a0520d..3d574e16de45 100644
@@ -2420,7 +2420,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 #endif /* CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT */
 
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
-static u64 kvm_supported_mem_attributes(struct kvm *kvm)
+u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
 	if (!kvm || kvm_arch_has_private_mem(kvm))
 		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
Signature · 1 line
2.52.0
author maintainer
#13 lore raw

PATCH 11/31 KVM: pass kvm == NULL case to kvm_arch_has_private_mem

In reply to Paolo Bonzini

Allow the architecture-specific code to enable CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES without exposing KVM_MEMORY_ATTRIBUTE_PRIVATE. This is mostly for consistency after introducing memory protection attributes; non-x86 architectures might want to add support for protection attributes without having KVM_MEMORY_ATTRIBUTE_PRIVATE show up in KVM_CHECK_EXTENSION.

---
 arch/x86/include/asm/kvm_host.h | 2 +-
 virt/kvm/kvm_main.c             | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
index c08781bb0327..c619552740a8 100644
@@ -1853,7 +1853,7 @@ enum kvm_intr_type {
 	 (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI)))
 
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
-#define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem)
+#define kvm_arch_has_private_mem(kvm) (!(kvm) || (kvm)->arch.has_private_mem)
 #endif
 
 #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state)
virt/kvm/kvm_main.c +1 -1
index 3d574e16de45..9d1836382d19 100644
@@ -2422,7 +2422,7 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
-	if (!kvm || kvm_arch_has_private_mem(kvm))
+	if (kvm_arch_has_private_mem(kvm))
 		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
 
 	return 0;
Signature · 1 line
2.52.0
author maintainer
#14 lore raw

PATCH 12/31 KVM: adjust for presence of more than one attribute

In reply to Paolo Bonzini

Remove or adjust code that assumed that the only valid attribute was KVM_MEMORY_ATTRIBUTE_PRIVATE, in preparation for the introduction of RWX memory attributes.

In particular, the optimization in kvm_range_has_memory_attributes simply is not valid if attrs == 0 but some of the attributes are being ignored. The WARNs in arch/x86/kvm/mmu/mmu.c are just a duplicate of checking kvm_supported_mem_attributes() during the ioctl, so drop it.

---
 arch/x86/kvm/mmu/mmu.c | 9 +--------
 virt/kvm/kvm_main.c    | 6 ++++--
 2 files changed, 5 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +1 -8
index ba462c671f7d..77d9c62d2a07 100644
@@ -8140,9 +8140,6 @@ bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 	 * Zapping SPTEs in this case ensures KVM will reassess whether or not
 	 * a hugepage can be used for affected ranges.
 	 */
-	if (WARN_ON_ONCE(!kvm_arch_has_private_mem(kvm)))
-		return false;
-
 	if (WARN_ON_ONCE(range->end <= range->start))
 		return false;
 
@@ -8218,11 +8215,7 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 	 * can't map memory PRIVATE.  KVM mustn't create a SHARED hugepage over
 	 * a range that has PRIVATE GFNs, and conversely converting a range to
 	 * SHARED may now allow hugepages.
-	 */
-	if (WARN_ON_ONCE(!kvm_arch_has_private_mem(kvm)))
-		return false;
-
-	/*
+	 *
 	 * The sequence matters here: upper levels consume the result of lower
 	 * level's scanning.
 	 */
virt/kvm/kvm_main.c +4 -2
index 9d1836382d19..85fdd33c51ab 100644
@@ -2436,10 +2436,12 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 				     unsigned long mask, unsigned long attrs)
 {
 	XA_STATE(xas, &kvm->mem_attr_array, start);
+	unsigned long supported_attrs;
 	unsigned long index;
 	void *entry;
 
-	mask &= kvm_supported_mem_attributes(kvm);
+	supported_attrs = kvm_supported_mem_attributes(kvm);
+	mask &= supported_attrs;
 	if (attrs & ~mask)
 		return false;
 
@@ -2447,7 +2449,7 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		return (kvm_get_memory_attributes(kvm, start) & mask) == attrs;
 
 	guard(rcu)();
-	if (!attrs)
+	if (!attrs && mask == supported_attrs)
 		return !xas_find(&xas, end - 1);
 
 	for (index = start; index < end; index++) {
Signature · 1 line
2.52.0
author maintainer
#15 lore raw

PATCH 13/31 KVM: Introduce NR/NW/NX memory attributes

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Introduce memory attributes to map guest physical memory regions as non-readable, non-writable, and/or non-executable. Only a subset of flag combinations is supported. Notably write and exec permissions require read permission, and memory protection attributes are incompatible with private memory.

As mentioned in 5a475554db1e ("KVM: Introduce per-page memory attributes", 2023-11-13), bits 0-2 of the memory attributes were reserved for RWX protection; they are negated to support current memory attribute users which use 0 to indicate no special treatment. Since 0 is not available, a non-negated version of the flags would need an extra bit to express no-access (R=0/W=0/X=0) mappings.

Unfortunately this precaution did not age too well; KVM now supports MBEC/GMET and adding mode-based memory protections will require a non-contiguous bit. But that's something left for later.

Different architectures may have different limitations on the set of valid protections, for example execution-only and XU=0 mappings are supported by Intel but not AMD processors[1]. So, add an architecture-specific callback and add a basic implementation for x86.

[1] When adding support for MBEC/GMET, since NX would remain to mean no execution at all, it is possible to use either an NXS bit or two separate NXS/NXU bits in addition to NX. The former would only support permissions that are available with either MBEC or GMET.

---
 Documentation/virt/kvm/api.rst  | 14 ++++++--
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/Kconfig            |  4 +--
 arch/x86/kvm/mmu/mmu.c          | 63 ++++++++++++++++++++++++---------
 arch/x86/kvm/x86.c              |  2 --
 include/linux/kvm_host.h        | 28 ++++++++++++++-
 include/uapi/linux/kvm.h        |  3 ++
 virt/kvm/kvm_main.c             | 36 +++++++++++++++----
 8 files changed, 119 insertions(+), 32 deletions(-)
Documentation/virt/kvm/api.rst +11 -3
index abc2ff1f8c84..81126ac11bf8 100644
@@ -6414,15 +6414,23 @@ of guest physical memory.
 	__u64 flags;
   };
 
+  #define KVM_MEMORY_ATTRIBUTE_NR                (1ULL << 0)
+  #define KVM_MEMORY_ATTRIBUTE_NW                (1ULL << 1)
+  #define KVM_MEMORY_ATTRIBUTE_NX                (1ULL << 2)
   #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
 
 The address and size must be page aligned.  The supported attributes can be
 retrieved via ioctl(KVM_CHECK_EXTENSION) on KVM_CAP_MEMORY_ATTRIBUTES.  If
 executed on a VM, KVM_CAP_MEMORY_ATTRIBUTES precisely returns the attributes
 supported by that VM.  If executed at system scope, KVM_CAP_MEMORY_ATTRIBUTES
-returns all attributes supported by KVM.  The only attribute defined at this
-time is KVM_MEMORY_ATTRIBUTE_PRIVATE, which marks the associated gfn as being
-guest private memory.
+returns all attributes supported by KVM.  The attribute defined at this
+time are:
+
+ - KVM_MEMORY_ATTRIBUTE_NR/NW/NX - Respectively marks the memory region as
+   non-read, non-write and/or non-exec.  Note that write-only, exec-only and
+   write-exec mappings are not supported.
+ - KVM_MEMORY_ATTRIBUTE_PRIVATE - Which marks the associated gfn as being guest
+   private memory.
 
 Note, there is no "get" API.  Userspace is responsible for explicitly tracking
 the state of a gfn/page as needed.
arch/x86/include/asm/kvm_host.h +1 -0
index c619552740a8..52c432f9bc66 100644
@@ -1853,6 +1853,7 @@ enum kvm_intr_type {
 	 (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI)))
 
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#define kvm_arch_has_memory_protection_attributes(kvm) (!(kvm) || !(kvm)->arch.has_private_mem)
 #define kvm_arch_has_private_mem(kvm) (!(kvm) || (kvm)->arch.has_private_mem)
 #endif
 
arch/x86/kvm/Kconfig +1 -3
index 538ed1e80332..4c630fd64fd2 100644
@@ -48,6 +48,7 @@ config KVM_X86
 	select KVM_GENERIC_PRE_FAULT_MEMORY
 	select KVM_WERROR if WERROR
 	select KVM_GUEST_MEMFD if X86_64
+	select KVM_GENERIC_MEMORY_ATTRIBUTES
 
 config KVM
 	tristate "Kernel-based Virtual Machine (KVM) support"
@@ -84,7 +85,6 @@ config KVM_SW_PROTECTED_VM
 	bool "Enable support for KVM software-protected VMs"
 	depends on EXPERT
 	depends on KVM_X86 && X86_64
-	select KVM_GENERIC_MEMORY_ATTRIBUTES
 	help
 	  Enable support for KVM software-protected VMs.  Currently, software-
 	  protected VMs are purely a development and testing vehicle for
@@ -135,7 +135,6 @@ config KVM_INTEL_TDX
 	bool "Intel Trust Domain Extensions (TDX) support"
 	default y
 	depends on INTEL_TDX_HOST
-	select KVM_GENERIC_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_POPULATE
 	help
 	  Provides support for launching Intel Trust Domain Extensions (TDX)
@@ -159,7 +158,6 @@ config KVM_AMD_SEV
 	depends on KVM_AMD && X86_64
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
-	select KVM_GENERIC_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_CONVERT
 	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
arch/x86/kvm/mmu/mmu.c +47 -16

89 lines not on this page · raw message

arch/x86/kvm/x86.c +0 -2
index 3338d85c721f..182fa31a286c 100644
@@ -10168,9 +10168,7 @@ static int kvm_alloc_memslot_metadata(struct kvm *kvm,
 		}
 	}
 
-#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 	kvm_mmu_init_memslot_memory_attributes(kvm, slot);
-#endif
 
 	if (kvm_page_track_create_memslot(kvm, slot, npages))
 		goto out_free;
include/linux/kvm_host.h +27 -1
index acffb2f79b3e..c215e3bdf043 100644
@@ -722,7 +722,9 @@ static inline int kvm_arch_vcpu_memslots_id(struct kvm_vcpu *vcpu)
 }
 #endif
 
-#ifndef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+bool kvm_arch_mem_attributes_supported_prot(struct kvm *kvm, unsigned long attrs);
+#else
 static inline bool kvm_arch_has_private_mem(struct kvm *kvm)
 {
 	return false;
@@ -2567,7 +2569,25 @@ static inline bool kvm_memslot_is_gmem_only(const struct kvm_memory_slot *slot)
 	return slot->flags & KVM_MEMSLOT_GMEM_ONLY;
 }
 
+static inline bool kvm_mem_attributes_may_read(u64 attrs)
+{
+	return !(attrs & KVM_MEMORY_ATTRIBUTE_NR);
+}
+
+static inline bool kvm_mem_attributes_may_write(u64 attrs)
+{
+	return !(attrs & KVM_MEMORY_ATTRIBUTE_NW);
+}
+
+static inline bool kvm_mem_attributes_may_exec(u64 attrs)
+{
+	return !(attrs & KVM_MEMORY_ATTRIBUTE_NX);
+}
+
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
+#define KVM_MEMORY_ATTRIBUTE_PROT \
+	(KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW | KVM_MEMORY_ATTRIBUTE_NX)
+
 static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
 	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
@@ -2579,6 +2599,7 @@ bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 					struct kvm_gfn_range *range);
 bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 					 struct kvm_gfn_range *range);
+bool kvm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs);
 u64 kvm_supported_mem_attributes(struct kvm *kvm);
 
 static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
@@ -2586,6 +2607,11 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 	return kvm_get_memory_attributes(kvm, gfn) & KVM_MEMORY_ATTRIBUTE_PRIVATE;
 }
 #else
+static inline bool kvm_mem_attributes_valid(struct kvm *kvm,
+					       unsigned long attrs)
+{
+	return false;
+}
 static inline u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
 	return 0;
include/uapi/linux/kvm.h +3 -0
index c23f1ea62eaf..ffbc68a6e2c9 100644
@@ -1657,6 +1657,9 @@ struct kvm_memory_attributes {
 	__u64 flags;
 };
 
+#define KVM_MEMORY_ATTRIBUTE_NR		       (1ULL << 0)
+#define KVM_MEMORY_ATTRIBUTE_NW		       (1ULL << 1)
+#define KVM_MEMORY_ATTRIBUTE_NX		       (1ULL << 2)
 #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
 
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
virt/kvm/kvm_main.c +29 -7
index 85fdd33c51ab..da80ce6b380e 100644
@@ -2422,10 +2422,15 @@ static int kvm_vm_ioctl_clear_dirty_log(struct kvm *kvm,
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
 u64 kvm_supported_mem_attributes(struct kvm *kvm)
 {
-	if (kvm_arch_has_private_mem(kvm))
-		return KVM_MEMORY_ATTRIBUTE_PRIVATE;
+	u64 supported_attrs = 0;
 
-	return 0;
+	if (kvm_arch_has_memory_protection_attributes(kvm))
+		supported_attrs |= KVM_MEMORY_ATTRIBUTE_PROT;
+
+	if (kvm_arch_has_private_mem(kvm))
+		supported_attrs |= KVM_MEMORY_ATTRIBUTE_PRIVATE;
+
+	return supported_attrs;
 }
 
 /*
@@ -2522,9 +2527,7 @@ static bool kvm_pre_set_memory_attributes(struct kvm *kvm,
 {
 	/*
 	 * Unconditionally add the range to the invalidation set, regardless of
-	 * whether or not the arch callback actually needs to zap SPTEs.  E.g.
-	 * if KVM supports RWX attributes in the future and the attributes are
-	 * going from R=>RW, zapping isn't strictly necessary.  Unconditionally
+	 * whether or not the arch callback actually needs to zap SPTEs.  Always
 	 * adding the range allows KVM to require that MMU invalidations add at
 	 * least one range between begin() and end(), e.g. allows KVM to detect
 	 * bugs where the add() is missed.  Relaxing the rule *might* be safe,
@@ -2599,6 +2602,25 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 
 	return r;
 }
+
+bool __weak kvm_arch_mem_attributes_supported_prot(struct kvm *kvm, unsigned long attrs)
+{
+	WARN_ONCE(true, "KVM_MEMORY_ATTRIBUTE_PROT requires kvm_arch_mem_attributes_supported_prot()");
+	return false;
+}
+
+bool kvm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs)
+{
+	if (attrs & ~kvm_supported_mem_attributes(kvm))
+		return false;
+
+	if ((attrs & KVM_MEMORY_ATTRIBUTE_PROT) &&
+	    !kvm_arch_mem_attributes_supported_prot(kvm, attrs))
+		return false;
+
+	return true;
+}
+
 static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 					   struct kvm_memory_attributes *attrs)
 {
@@ -2607,7 +2629,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,
 	/* flags is currently not used. */
 	if (attrs->flags)
 		return -EINVAL;
-	if (attrs->attributes & ~kvm_supported_mem_attributes(kvm))
+	if (!kvm_mem_attributes_valid(kvm, attrs->attributes))
 		return -EINVAL;
 	if (attrs->size == 0 || attrs->address + attrs->size < attrs->address)
 		return -EINVAL;
Signature · 1 line
2.52.0
author maintainer
#16 lore raw

PATCH 14/31 KVM: Include memory protections in result of gfn->hva conversion

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

All paths that lead to guest memory accesses now need to check whether memory attributes allow that access. Users of gfn_to_hva, kvm_vcpu_gfn_to_hva and their *_prot variant can get it more or less for free via erroneous return values, so do this first.

Note however that this is not true of the variants that take a cached kvm_memslots pointer. These include caches (gfn-to-hva and gfn-to-pfn) and page faults, both of which will need specific changes; but the more optimized functions in virt/kvm/kvm_main.c such as kvm_read_guest() and kvm_write_guest() also retrieve the memslot high in the call chain, and therefore they will need changes in __kvm_read/write_guest_page().

---
 include/linux/kvm_host.h | 18 +++++++++++++
 virt/kvm/kvm_main.c      | 55 +++++++++++++++++++++++++++++++++++++---
 2 files changed, 69 insertions(+), 4 deletions(-)
include/linux/kvm_host.h +18 -0
index c215e3bdf043..01223301f0e5 100644
@@ -2620,8 +2620,26 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)
 {
 	return false;
 }
+static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
+{
+	return 0;
+}
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
+static inline int kvm_mem_attributes_may_read_gfn(struct kvm *kvm, gfn_t gfn)
+{
+	unsigned long attrs = kvm_get_memory_attributes(kvm, gfn);
+
+	return kvm_mem_attributes_may_read(attrs);
+}
+
+static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
+{
+	unsigned long attrs = kvm_get_memory_attributes(kvm, gfn);
+
+	return kvm_mem_attributes_may_write(attrs);
+}
+
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
virt/kvm/kvm_main.c +51 -4

79 lines not on this page · raw message

Signature · 1 line
2.52.0
author maintainer
#17 lore raw

PATCH 15/31 KVM: Introduce kvm_fetch_guest_page() and use it for x86

In reply to Paolo Bonzini

Introduce a separate function so that the emulator will be able to take into account execution memory attributes rather than read.

---
 arch/x86/kvm/x86.c       | 4 ++--
 include/linux/kvm_host.h | 2 ++
 virt/kvm/kvm_main.c      | 9 +++++++++
 3 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +2 -2
index 182fa31a286c..e07966b9ef85 100644
@@ -4872,8 +4872,8 @@ static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt,
 	offset = addr & (PAGE_SIZE-1);
 	if (WARN_ON(offset + bytes > PAGE_SIZE))
 		bytes = (unsigned)PAGE_SIZE - offset;
-	ret = kvm_vcpu_read_guest_page(vcpu, gpa >> PAGE_SHIFT, val,
-				       offset, bytes);
+	ret = kvm_vcpu_fetch_guest_page(vcpu, gpa >> PAGE_SHIFT, val,
+					offset, bytes);
 	if (unlikely(ret < 0))
 		return X86EMUL_IO_NEEDED;
 
include/linux/kvm_host.h +2 -0
index 01223301f0e5..cddf0b943757 100644
@@ -1444,6 +1444,8 @@ DEFINE_VCPU_MAP_CLASS(_readonly);
 
 unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn);
 unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable);
+int kvm_vcpu_fetch_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
+			      int offset, int len);
 int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, int offset,
 			     int len);
 int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa, void *data,
virt/kvm/kvm_main.c +9 -0
index 3b43a0d2a5d8..9fdec95ccce2 100644
@@ -3289,6 +3289,15 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_page);
 
+int kvm_vcpu_fetch_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
+			      int offset, int len)
+{
+	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
+
+	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_fetch_guest_page);
+
 int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long len)
 {
 	gfn_t gfn = gpa >> PAGE_SHIFT;
Signature · 1 line
2.52.0
author maintainer
#18 lore raw

PATCH 16/31 KVM: Take memory protections into account for memory read/write/fetch

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Take into account memory attributes when accessing guest memory through the kvm_{read,write,fetch}*() family of functions.

All of these pass a struct kvm_memory_slot pointer to the actual workhorse functions, in order to share code between the VM-wide and vCPU-specific version of the functions (the latter of which handles the multi-address-space case). For this reason they need specific changes and do not work even though the gfn_to_hva() path has been taught already about memory protection attributes.

---
 virt/kvm/kvm_main.c | 34 ++++++++++++++++++++++++++--------
 1 file changed, 26 insertions(+), 8 deletions(-)
virt/kvm/kvm_main.c +26 -8
index 9fdec95ccce2..acf09059a1cd 100644
@@ -3253,11 +3253,13 @@ static int next_segment(unsigned long len, int offset)
 }
 
 /* Copy @len bytes from guest memory at '(@gfn * PAGE_SIZE) + @offset' to @data */
-static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
-				 void *data, int offset, int len)
+static int __kvm_read_guest_page(struct kvm *kvm, struct kvm_memory_slot *slot,
+				 gfn_t gfn, void *data, int offset, int len,
+				 unsigned long attr)
 {
 	int r;
 	unsigned long addr;
+	unsigned long attrs;
 
 	if (WARN_ON_ONCE(offset + len > PAGE_SIZE))
 		return -EFAULT;
@@ -3265,6 +3267,11 @@ static int __kvm_read_guest_page(struct kvm_memory_slot *slot, gfn_t gfn,
 	addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
+
+	attrs = kvm_get_memory_attributes(kvm, gfn);
+	if (attrs & attr)
+		return -EFAULT;
+
 	r = __copy_from_user(data, (void __user *)addr + offset, len);
 	if (r)
 		return -EFAULT;
@@ -3276,7 +3283,8 @@ int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset,
 {
 	struct kvm_memory_slot *slot = gfn_to_memslot(kvm, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(kvm, slot, gfn, data, offset, len,
+				     KVM_MEMORY_ATTRIBUTE_NR);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_read_guest_page);
 
@@ -3285,7 +3293,8 @@ int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len,
+				     KVM_MEMORY_ATTRIBUTE_NR);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_page);
 
@@ -3294,7 +3303,8 @@ int kvm_vcpu_fetch_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,
 {
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 
-	return __kvm_read_guest_page(slot, gfn, data, offset, len);
+	return __kvm_read_guest_page(vcpu->kvm, slot, gfn, data, offset, len,
+				     KVM_MEMORY_ATTRIBUTE_NX);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_fetch_guest_page);
 
@@ -3338,8 +3348,9 @@ int kvm_vcpu_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa, void *data, unsigned l
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest);
 
-static int __kvm_read_guest_atomic(struct kvm_memory_slot *slot, gfn_t gfn,
-			           void *data, int offset, unsigned long len)
+static int __kvm_read_guest_atomic(struct kvm *kvm,
+				   struct kvm_memory_slot *slot, gfn_t gfn,
+				   void *data, int offset, unsigned long len)
 {
 	int r;
 	unsigned long addr;
@@ -3347,6 +3358,9 @@ static int __kvm_read_guest_atomic(struct kvm_memory_slot *slot, gfn_t gfn,
 	if (WARN_ON_ONCE(offset + len > PAGE_SIZE))
 		return -EFAULT;
 
+	if (!kvm_mem_attributes_may_read_gfn(kvm, gfn))
+		return -EFAULT;
+
 	addr = gfn_to_hva_memslot_prot(slot, gfn, NULL);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
@@ -3365,7 +3379,7 @@ int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa,
 	struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
 	int offset = offset_in_page(gpa);
 
-	return __kvm_read_guest_atomic(slot, gfn, data, offset, len);
+	return __kvm_read_guest_atomic(vcpu->kvm, slot, gfn, data, offset, len);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_vcpu_read_guest_atomic);
 
@@ -3383,6 +3397,10 @@ static int __kvm_write_guest_page(struct kvm *kvm,
 	addr = gfn_to_hva_memslot(memslot, gfn);
 	if (kvm_is_error_hva(addr))
 		return -EFAULT;
+
+	if (!kvm_mem_attributes_may_write_gfn(kvm, gfn))
+		return -EFAULT;
+
 	r = __copy_to_user((void __user *)addr + offset, data, len);
 	if (r)
 		return -EFAULT;
Signature · 1 line
2.52.0
author maintainer
#19 lore raw

PATCH 17/31 KVM: Take memory protections into account for __kvm_vcpu_map

In reply to Paolo Bonzini

Take into account memory attributes when establishing a long term map through kvm_vcpu_map()/kvm_vcpu_map_readonly().

Unlike the gfn-to-pfn cache, changes to the attributes will not be propagated, so that it will be possible to place a nested VMCS, VMCB, or APIC access page on a non-readable/writable page. This will be fixed separately by using a gfn-to-pfn cache for nested virtualization data structures, in the meanwhile apply a stopgap fix.

---
 virt/kvm/kvm_main.c | 5 +++++
 1 file changed, 5 insertions(+)
virt/kvm/kvm_main.c +5 -0
index acf09059a1cd..1d111502b8c1 100644
@@ -3193,6 +3193,11 @@ int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map,
 	if (WARN_ON_ONCE(map->hva))
 		kvm_vcpu_unmap(vcpu, map);
 
+	if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gfn))
+		return -EFAULT;
+	if (writable && !kvm_mem_attributes_may_write_gfn(vcpu->kvm, gfn))
+		return -EFAULT;
+
 	map->pinned_page = NULL;
 	map->page = NULL;
 	map->hva = NULL;
Signature · 1 line
2.52.0
author maintainer
#20 lore raw

PATCH 18/31 KVM: Encapsulate memattrs array into anonymous struct

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

The metadata surrounding memory attributes is about to grow, so encapsulate the memory attributes array within an anonymous struct to provide namespacing.

---
 include/linux/kvm_host.h |  8 +++++---
 virt/kvm/kvm_main.c      | 10 +++++-----
 2 files changed, 10 insertions(+), 8 deletions(-)
include/linux/kvm_host.h +5 -3
index cddf0b943757..24d04322d89e 100644
@@ -875,8 +875,10 @@ struct kvm {
 	struct notifier_block pm_notifier;
 #endif
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
-	/* Protected by slots_lock (for writes) and RCU (for reads) */
-	struct xarray mem_attr_array;
+	struct {
+		/* Protected by slots_lock (for writes) and RCU (for reads) */
+		struct xarray array;
+	} mem_attrs;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
 };
@@ -2592,7 +2594,7 @@ static inline bool kvm_mem_attributes_may_exec(u64 attrs)
 
 static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
-	return xa_to_value(xa_load(&kvm->mem_attr_array, gfn));
+	return xa_to_value(xa_load(&kvm->mem_attrs.array, gfn));
 }
 
 bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
virt/kvm/kvm_main.c +5 -5
index 1d111502b8c1..b8e419fb6cbf 100644
@@ -1117,7 +1117,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname)
 	rcuwait_init(&kvm->mn_memslots_update_rcuwait);
 	xa_init(&kvm->vcpu_array);
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
-	xa_init(&kvm->mem_attr_array);
+	xa_init(&kvm->mem_attrs.array);
 #endif
 
 	INIT_LIST_HEAD(&kvm->gpc_list);
@@ -1302,7 +1302,7 @@ static void kvm_destroy_vm(struct kvm *kvm)
 	srcu_barrier(&kvm->srcu);
 	cleanup_srcu_struct(&kvm->srcu);
 #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES
-	xa_destroy(&kvm->mem_attr_array);
+	xa_destroy(&kvm->mem_attrs.array);
 #endif
 	kvm_arch_free_vm(kvm);
 	preempt_notifier_dec();
@@ -2440,7 +2440,7 @@ u64 kvm_supported_mem_attributes(struct kvm *kvm)
 bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 				     unsigned long mask, unsigned long attrs)
 {
-	XA_STATE(xas, &kvm->mem_attr_array, start);
+	XA_STATE(xas, &kvm->mem_attrs.array, start);
 	unsigned long supported_attrs;
 	unsigned long index;
 	void *entry;
@@ -2579,7 +2579,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	 * partway through setting the new attributes.
 	 */
 	for (i = start; i < end; i++) {
-		r = xa_reserve(&kvm->mem_attr_array, i, GFP_KERNEL_ACCOUNT);
+		r = xa_reserve(&kvm->mem_attrs.array, i, GFP_KERNEL_ACCOUNT);
 		if (r)
 			goto out_unlock;
 
@@ -2589,7 +2589,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	kvm_handle_gfn_range(kvm, &pre_set_range);
 
 	for (i = start; i < end; i++) {
-		r = xa_err(xa_store(&kvm->mem_attr_array, i, entry,
+		r = xa_err(xa_store(&kvm->mem_attrs.array, i, entry,
 				    GFP_KERNEL_ACCOUNT));
 		KVM_BUG_ON(r, kvm);
 		cond_resched();
Signature · 1 line
2.52.0
author maintainer
#21 lore raw

PATCH 19/31 KVM: loongarch: do full validity check on the gfn-to-hva cache

In reply to Paolo Bonzini

If the HVA or the memslots are invalid, reinitialize the cache.

---
 arch/loongarch/kvm/vcpu.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
arch/loongarch/kvm/vcpu.c +4 -2
index ed9e092c97ba..d540004d74ff 100644
@@ -165,7 +165,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	slots = kvm_memslots(vcpu->kvm);
-	if (slots->generation != ghc->generation || gpa != ghc->gpa) {
+	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
@@ -1811,7 +1812,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
 	slots = kvm_memslots(vcpu->kvm);
-	if (slots->generation != ghc->generation || gpa != ghc->gpa) {
+	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
Signature · 1 line
2.52.0
author maintainer
#22 lore raw

PATCH 20/31 KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()

In reply to Paolo Bonzini

In many cases, retrieving kvm_memslots is followed by a check on the generation of the slots. Introduce a helper function that either does the check alone, or compounds it with returning the struct kvm_memslots* to the caller.

---
 arch/loongarch/kvm/vcpu.c | 12 ++++--------
 arch/x86/kvm/x86.c        | 10 ++--------
 include/linux/kvm_host.h  | 13 +++++++++++++
 virt/kvm/kvm_main.c       |  8 ++++----
 virt/kvm/pfncache.c       | 10 ++++------
 5 files changed, 27 insertions(+), 26 deletions(-)
arch/loongarch/kvm/vcpu.c +4 -8
index d540004d74ff..cf4f4d75374a 100644
@@ -154,7 +154,6 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 	u32 version;
 	u64 steal;
 	gpa_t gpa;
-	struct kvm_memslots *slots;
 	struct kvm_steal_time __user *st;
 	struct gfn_to_hva_cache *ghc;
 
@@ -164,9 +163,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 		return;
 
 	gpa &= KVM_STEAL_PHYS_MASK;
-	slots = kvm_memslots(vcpu->kvm);
-	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
-		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
@@ -1794,7 +1792,6 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 {
 	gpa_t gpa;
 	struct gfn_to_hva_cache *ghc;
-	struct kvm_memslots *slots;
 	struct kvm_steal_time __user *st;
 
 	gpa = vcpu->arch.st.guest_addr;
@@ -1811,9 +1808,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
-	slots = kvm_memslots(vcpu->kvm);
-	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
-		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
arch/x86/kvm/x86.c +2 -8
index e07966b9ef85..7a1e80bb28f5 100644
@@ -2049,7 +2049,6 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 {
 	struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
 	struct kvm_steal_time __user *st;
-	struct kvm_memslots *slots;
 	gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
 	u64 steal;
 	u32 version;
@@ -2065,9 +2064,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	if (WARN_ON_ONCE(current->mm != vcpu->kvm->mm))
 		return;
 
-	slots = kvm_memslots(vcpu->kvm);
-
-	if (unlikely(slots->generation != ghc->generation ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		/* We rely on the fact that it fits in a single page. */
@@ -2617,7 +2614,6 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
 {
 	struct gfn_to_hva_cache *ghc = &vcpu->arch.st.cache;
 	struct kvm_steal_time __user *st;
-	struct kvm_memslots *slots;
 	static const u8 preempted = KVM_VCPU_PREEMPTED;
 	gpa_t gpa = vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS;
 
@@ -2644,9 +2640,7 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
 	if (unlikely(current->mm != vcpu->kvm->mm))
 		return;
 
-	slots = kvm_memslots(vcpu->kvm);
-
-	if (unlikely(slots->generation != ghc->generation ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot))
 		return;
include/linux/kvm_host.h +13 -0
index 24d04322d89e..7599ab7ad7ef 100644
@@ -2644,6 +2644,19 @@ static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
 	return kvm_mem_attributes_may_write(attrs);
 }
 
+static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation, struct kvm_memslots **p_slots)
+{
+	struct kvm_memslots *slots = *p_slots = kvm_memslots(kvm);
+	return slots->generation == slots_generation;
+}
+
+static inline bool kvm_check_gen(struct kvm *kvm, u64 slots_generation)
+{
+	struct kvm_memslots *slots;
+	return kvm_memslots_check_gen(kvm, slots_generation, &slots);
+}
+
+
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
virt/kvm/kvm_main.c +4 -4
index b8e419fb6cbf..ea4e9425a82d 100644
@@ -3526,14 +3526,14 @@ int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 				  void *data, unsigned int offset,
 				  unsigned long len)
 {
-	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memslots *slots;
 	int r;
 	gpa_t gpa = ghc->gpa + offset;
 
 	if (WARN_ON_ONCE(len + offset > ghc->len))
 		return -EINVAL;
 
-	if (slots->generation != ghc->generation) {
+	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->generation, &slots))) {
 		if (__kvm_gfn_to_hva_cache_init(slots, ghc, ghc->gpa, ghc->len))
 			return -EFAULT;
 	}
@@ -3564,14 +3564,14 @@ int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 				 void *data, unsigned int offset,
 				 unsigned long len)
 {
-	struct kvm_memslots *slots = kvm_memslots(kvm);
+	struct kvm_memslots *slots;
 	int r;
 	gpa_t gpa = ghc->gpa + offset;
 
 	if (WARN_ON_ONCE(len + offset > ghc->len))
 		return -EINVAL;
 
-	if (slots->generation != ghc->generation) {
+	if (unlikely(!kvm_memslots_check_gen(kvm, ghc->generation, &slots))) {
 		if (__kvm_gfn_to_hva_cache_init(slots, ghc, ghc->gpa, ghc->len))
 			return -EFAULT;
 	}
virt/kvm/pfncache.c +4 -6
index 728d2c1b488a..e09703d249bb 100644
@@ -72,8 +72,6 @@ static bool kvm_gpc_is_valid_len(gpa_t gpa, unsigned long uhva,
 
 bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
 {
-	struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
-
 	if (!gpc->active)
 		return false;
 
@@ -81,7 +79,7 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
 	 * If the page was cached from a memslot, make sure the memslots have
 	 * not been re-configured.
 	 */
-	if (!kvm_is_error_gpa(gpc->gpa) && gpc->generation != slots->generation)
+	if (!kvm_is_error_gpa(gpc->gpa) && !kvm_check_gen(gpc->kvm, gpc->generation))
 		return false;
 
 	if (kvm_is_error_hva(gpc->uhva))
@@ -290,12 +288,12 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
 		if (gpc->uhva != old_uhva)
 			hva_change = true;
 	} else {
-		struct kvm_memslots *slots = kvm_memslots(gpc->kvm);
+		struct kvm_memslots *slots;
 
 		page_offset = offset_in_page(gpa);
 
-		if (gpc->gpa != gpa || gpc->generation != slots->generation ||
-		    kvm_is_error_hva(gpc->uhva)) {
+		if (!kvm_memslots_check_gen(gpc->kvm, gpc->generation, &slots) ||
+		    gpc->gpa != gpa || kvm_is_error_hva(gpc->uhva)) {
 			gfn_t gfn = gpa_to_gfn(gpa);
 
 			gpc->gpa = gpa;
Signature · 1 line
2.52.0
#23 lore raw

Re: [PATCH 20/31] KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] kvm: missing closing parenthesis for unlikely() macro in loongarch --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] kvm: missing closing parenthesis for unlikely() macro in loongarch --

commit 173c930f2b4596e5daa23aa65be9736878ddf068
Author: Paolo Bonzini <[email protected]>

KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()

This commit introduces helper functions to retrieve and check the generation of kvm_memslots. It replaces direct accesses to the generation field with kvm_check_gen() or kvm_memslots_check_gen() across multiple architectures.

Quoted 4 lines
arch/loongarch/kvm/vcpu.c +0 -0
index d540004d74ff6..cf4f4d75374ad 100644

[ ... ]

arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 167

Quoted 9 lines
arch/loongarch/kvm/vcpu.c +2 -3
@@ -164,9 +163,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 		return;
 
 	gpa &= KVM_STEAL_PHYS_MASK;
-	slots = kvm_memslots(vcpu->kvm);
-	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
-		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {

[Severity: Low]
Does this missing closing parenthesis for the unlikely() macro cause a build failure on LoongArch architectures? It looks like we have !ghc->memslot) { instead of !ghc->memslot)) { here in kvm_update_stolen_time().

Quoted 3 lines
arch/loongarch/kvm/vcpu.c +0 -0
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;

[ ... ]

arch/loongarch/kvm/vcpu.c static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu) · line 1812

Quoted 9 lines
arch/loongarch/kvm/vcpu.c +2 -3
@@ -1811,9 +1808,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
-	slots = kvm_memslots(vcpu->kvm);
-	if (unlikely(slots->generation != ghc->generation || gpa != ghc->gpa ||
-		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {

[Severity: Low]
Is this the same syntax issue as above? It seems we are also missing the closing parenthesis for unlikely() in kvm_vcpu_set_pv_preempted().

Quoted 3 lines
arch/loongarch/kvm/vcpu.c +0 -0
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
author maintainer
#24 lore raw

PATCH 21/31 KVM: Introduce a generation number for memory attributes

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Introduce a generation number to track memory attribute modifications. This will allow KVM components to invalidate any assumptions they might have about guest's physical addresses and their permissions when memory attributes change.

Like with memory slot updates, it's mandatory for components that access guest memory based on cached information to do so within a KVM SRCU read-side critical section, and that they validate the generation number before accessing memory. This, in combination with the synchronize_srcu() call within the memory attributes ioctl handler, ensures the following:
- A memory attribute modification operation only returns after all users of outdated GPA data are done running.
- Any component accessing cached data after the memory attribute modification returned will see the updated generation number.

Additionally, loads/stores of the generation number have acquire/release semantics; which ensures all attribute writes are visible before updating the generation, and loads from attributes happen after having read the current generation number.

Ultimately, since synchronize_srcu_expedited() is an expensive operation, only perform it when absolutely necessary. Do so if the introduced memory attribute is known to require synchronization or if the attribute being cleared contained a memory attribute that required synchronization.

There shouldn't be any performance loss for memory attributes that don't require synchronization (and in general for any VMM that does not apply memory protections), because the attributes generation will always remain unchanged.

---
 arch/loongarch/kvm/vcpu.c  |  6 ++-
 arch/x86/kvm/x86.c         |  4 +-
 include/linux/kvm_host.h   | 34 ++++++++++++++--
 include/linux/kvm_types.h  |  6 ++-
 include/trace/events/kvm.h | 14 +++++--
 virt/kvm/kvm_main.c        | 80 +++++++++++++++++++++++++++++++++-----
 virt/kvm/pfncache.c        | 13 ++++---
 7 files changed, 128 insertions(+), 29 deletions(-)
arch/loongarch/kvm/vcpu.c +4 -2
index cf4f4d75374a..1179c413b048 100644
@@ -163,7 +163,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 		return;
 
 	gpa &= KVM_STEAL_PHYS_MASK;
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
@@ -1808,7 +1809,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
arch/x86/kvm/x86.c +2 -2
index 7a1e80bb28f5..ab4f5e44d74d 100644
@@ -2064,7 +2064,7 @@ static void record_steal_time(struct kvm_vcpu *vcpu)
 	if (WARN_ON_ONCE(current->mm != vcpu->kvm->mm))
 		return;
 
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot)) {
 		/* We rely on the fact that it fits in a single page. */
@@ -2640,7 +2640,7 @@ static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu)
 	if (unlikely(current->mm != vcpu->kvm->mm))
 		return;
 
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
 		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot))
 		return;
include/linux/kvm_host.h +30 -4
index 7599ab7ad7ef..ab62ce37bc18 100644
@@ -878,6 +878,7 @@ struct kvm {
 	struct {
 		/* Protected by slots_lock (for writes) and RCU (for reads) */
 		struct xarray array;
+		u64 generation;
 	} mem_attrs;
 #endif
 	char stats_id[KVM_STATS_NAME_SIZE];
@@ -2592,13 +2593,31 @@ static inline bool kvm_mem_attributes_may_exec(u64 attrs)
 #define KVM_MEMORY_ATTRIBUTE_PROT \
 	(KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW | KVM_MEMORY_ATTRIBUTE_NX)
 
+#define KVM_MEMORY_ATTRIBUTE_NEEDS_SYNC_MASK KVM_MEMORY_ATTRIBUTE_PROT
+
 static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn)
 {
 	return xa_to_value(xa_load(&kvm->mem_attrs.array, gfn));
 }
 
+static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
+{
+	RCU_LOCKDEP_WARN(!lockdep_is_held(&kvm->slots_lock) &&
+				 !srcu_read_lock_held(&kvm->srcu),
+			 "Suspicious memory attribute generation usage\n");
+
+	/*
+	 * The acquire pairs with the release in kvm_vm_set_mem_attributes().
+	 * Memory attributes should only be queried _after_ storing the
+	 * generation number.
+	 */
+	return smp_load_acquire(&kvm->mem_attrs.generation);
+}
+
 bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 				     unsigned long mask, unsigned long attrs);
+bool kvm_range_has_any_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
+					unsigned long mask);
 bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 					struct kvm_gfn_range *range);
 bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
@@ -2628,6 +2647,10 @@ static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn
 {
 	return 0;
 }
+static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
+{
+	return 0;
+}
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
 static inline int kvm_mem_attributes_may_read_gfn(struct kvm *kvm, gfn_t gfn)
@@ -2644,16 +2667,19 @@ static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
 	return kvm_mem_attributes_may_write(attrs);
 }
 
-static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation, struct kvm_memslots **p_slots)
+static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation,
+					  u64 attrs_generation, struct kvm_memslots **p_slots)
 {
 	struct kvm_memslots *slots = *p_slots = kvm_memslots(kvm);
-	return slots->generation == slots_generation;
+	return slots->generation == slots_generation &&
+		kvm_mem_attributes_generation(kvm) == attrs_generation;
 }
 
-static inline bool kvm_check_gen(struct kvm *kvm, u64 slots_generation)
+static inline bool kvm_check_gen(struct kvm *kvm, u64 slots_generation,
+				 u64 attrs_generation)
 {
 	struct kvm_memslots *slots;
-	return kvm_memslots_check_gen(kvm, slots_generation, &slots);
+	return kvm_memslots_check_gen(kvm, slots_generation, attrs_generation, &slots);
 }
 
 
include/linux/kvm_types.h +4 -2
index a568d8e6f4e8..7d911220e00d 100644
@@ -74,7 +74,8 @@ typedef u64            hfn_t;
 typedef hfn_t kvm_pfn_t;
 
 struct gfn_to_hva_cache {
-	u64 generation;
+	u64 slots_generation;
+	u64 attrs_generation;
 	gpa_t gpa;
 	unsigned long hva;
 	unsigned long len;
@@ -82,7 +83,8 @@ struct gfn_to_hva_cache {
 };
 
 struct gfn_to_pfn_cache {
-	u64 generation;
+	u64 slots_generation;
+	u64 attrs_generation;
 	gpa_t gpa;
 	unsigned long uhva;
 	struct kvm_memory_slot *memslot;
include/trace/events/kvm.h +10 -4
index b282e3a86769..a620131e9010 100644
@@ -365,23 +365,29 @@ TRACE_EVENT(kvm_dirty_ring_exit,
  * @attr:	The value of the attribute being set.
  */
 TRACE_EVENT(kvm_vm_set_mem_attributes,
-	TP_PROTO(gfn_t start, gfn_t end, unsigned long attr),
-	TP_ARGS(start, end, attr),
+	TP_PROTO(gfn_t start, gfn_t end, unsigned long attr, bool sync, u64 generation),
+	TP_ARGS(start, end, attr, sync, generation),
 
 	TP_STRUCT__entry(
 		__field(gfn_t,		start)
 		__field(gfn_t,		end)
 		__field(unsigned long,	attr)
+		__field(bool,		sync)
+		__field(u64,		generation)
 	),
 
 	TP_fast_assign(
 		__entry->start		= start;
 		__entry->end		= end;
 		__entry->attr		= attr;
+		__entry->sync		= sync;
+		__entry->generation	= generation;
 	),
 
-	TP_printk("%#016llx -- %#016llx [0x%lx]",
-		  __entry->start, __entry->end, __entry->attr)
+	TP_printk("%#016llx -- %#016llx [0x%lx], sync %d gen %llu",
+		  __entry->start, __entry->end, __entry->attr,
+		  __entry->sync, __entry->generation)
+
 );
 #endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
 
virt/kvm/kvm_main.c +70 -10

156 lines not on this page · raw message

virt/kvm/pfncache.c +8 -5
index e09703d249bb..46ffae69fe77 100644
@@ -76,10 +76,11 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len)
 		return false;
 
 	/*
-	 * If the page was cached from a memslot, make sure the memslots have
-	 * not been re-configured.
+	 * If the page was cached from a memslot, make sure the memslots nor
+	 * memory attributes have not been re-configured.
 	 */
-	if (!kvm_is_error_gpa(gpc->gpa) && !kvm_check_gen(gpc->kvm, gpc->generation))
+	if (!kvm_is_error_gpa(gpc->gpa) &&
+	    !kvm_check_gen(gpc->kvm, gpc->slots_generation, gpc->attrs_generation))
 		return false;
 
 	if (kvm_is_error_hva(gpc->uhva))
@@ -253,6 +254,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc)
 
 static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned long uhva)
 {
+	struct kvm *kvm = gpc->kvm;
 	unsigned long page_offset;
 	bool unmap_old = false;
 	unsigned long old_uhva;
@@ -292,12 +294,13 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
 
 		page_offset = offset_in_page(gpa);
 
-		if (!kvm_memslots_check_gen(gpc->kvm, gpc->generation, &slots) ||
+		if (!kvm_memslots_check_gen(gpc->kvm, gpc->slots_generation, gpc->attrs_generation, &slots) ||
 		    gpc->gpa != gpa || kvm_is_error_hva(gpc->uhva)) {
 			gfn_t gfn = gpa_to_gfn(gpa);
 
+			gpc->attrs_generation = kvm_mem_attributes_generation(kvm);
 			gpc->gpa = gpa;
-			gpc->generation = slots->generation;
+			gpc->slots_generation = slots->generation;
 			gpc->memslot = __gfn_to_memslot(slots, gfn);
 			gpc->uhva = gfn_to_hva_memslot(gpc->memslot, gfn);
 
Signature · 1 line
2.52.0
#25 lore raw

Re: [PATCH 21/31] KVM: Introduce a generation number for memory attributes

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] kvm: infinite loop in mem_attributes checking due to xas_next() misuse - [Low] kvm: loongarch…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] kvm: infinite loop in mem_attributes checking due to xas_next() misuse
- [Low] kvm: loongarch: build failure due to missing closing parenthesis in unlikely() --

commit ca19ec78b0ccf685019346276e3422a33ce7e2e0
Author: Nicolas Saenz Julienne <[email protected]>

KVM: Introduce a generation number for memory attributes

This commit introduces a generation number to track memory attribute modifications in KVM. It ensures components accessing guest memory via cached information validate the generation within a KVM SRCU read-side critical section.

arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 168

Quoted 11 lines
arch/loongarch/kvm/vcpu.c +2 -1
@@ -163,7 +163,8 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 		return;
 
 	gpa &= KVM_STEAL_PHYS_MASK;
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {

[Severity: Low]
Does this missing closing parenthesis for unlikely cause a build regression?

It looks like the macro is left unclosed, which breaks compilation on LoongArch.

[ ... ]

arch/loongarch/kvm/vcpu.c static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu) · line 1814

Quoted 8 lines
arch/loongarch/kvm/vcpu.c +2 -1
@@ -1808,7 +1809,8 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
-	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->generation) || gpa != ghc->gpa ||
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
 		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {

[Severity: Low]
Does this missing closing parenthesis for unlikely cause a build regression here too?

Quoted 3 lines
virt/kvm/kvm_main.c +0 -0

[ ... ]

Quoted 5 lines
virt/kvm/kvm_main.c +5 -0
+bool kvm_range_has_any_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
+					unsigned long mask)
+{
+	XA_STATE(xas, &kvm->mem_attrs.array, start);
+	void *entry;

[ ... ]

virt/kvm/kvm_main.c bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end, · line 2498

Quoted 8 lines
virt/kvm/kvm_main.c +8 -0
+	guard(rcu)();
+	for (;;) {
+		do {
+			entry = xas_next(&xas);
+		} while (xas_retry(&xas, entry));
+
+		if (xas.xa_index >= end)
+			break;

[Severity: High]
Can this loop regression lead to an infinite loop if the queried range extends beyond the populated entries of the XArray?

When xas_next() returns NULL and transitions the state to XAS_BOUNDS, subsequent calls to xas_next() do not increment xas.xa_index. Because xa_index stops advancing, the termination condition xas.xa_index >= end is never met.

This can cause a hard lockup inside an RCU read-side critical section when reachable via untrusted userspace ioctls.

The same xas_next() misuse appears to be present in kvm_range_has_memory_attributes() which can cause it to incorrectly return false for ranges beyond populated bounds due to the xas.xa_index != index check.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=21
author maintainer
#26 lore raw

PATCH 22/31 KVM: Take memory protections into account for accesses with cached gfn->hva

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Account for memory attributes when accessing guest memory through kvm_get/put_guest().

This requires tracking the memory attributes generation as part of gfn_to_hva_cache's data, invalidate the cached information if the generation changes, and failing to refresh the cache if restrictive memory attributes are found within the GPA range.

Similar to how gfn_to_hva_cache disallows caching gfns mapped within read-only memory slots, gfns marked as read-only by memory attributes will also fail to initialize. Unsurprisingly, the same behaviour applies to gfns mapped as non-accessible (NR/NW), while non-executable mappings are okay.

---
 include/linux/kvm_host.h | 20 +++++++++++++++++---
 virt/kvm/kvm_main.c      | 15 +++++++++++----
 2 files changed, 28 insertions(+), 7 deletions(-)
include/linux/kvm_host.h +17 -3
index ab62ce37bc18..e9c0932f150e 100644
@@ -1359,7 +1359,8 @@ int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	typeof(v) __user *__uaddr = (typeof(__uaddr))(__addr + offset);	\
 	int __ret = -EFAULT;						\
 									\
-	if (!kvm_is_error_hva(__addr))					\
+	if (!kvm_is_error_hva(__addr) &&                                \
+	    kvm_mem_attributes_may_read_gfn(kvm, gfn))		\
 		__ret = get_user(v, __uaddr);				\
 	__ret;								\
 })
@@ -1379,7 +1380,8 @@ int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *ghc,
 	typeof(v) __user *__uaddr = (typeof(__uaddr))(__addr + offset);	\
 	int __ret = -EFAULT;						\
 									\
-	if (!kvm_is_error_hva(__addr))					\
+	if (!kvm_is_error_hva(__addr) &&                                \
+	    kvm_mem_attributes_may_write_gfn(kvm, gfn))		\
 		__ret = put_user(v, __uaddr);				\
 	if (!__ret)							\
 		mark_page_dirty(kvm, gfn);				\
@@ -2647,6 +2649,12 @@ static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn
 {
 	return 0;
 }
+static inline bool kvm_range_has_any_memory_attributes(struct kvm *kvm,
+						       gfn_t start, gfn_t end,
+						       unsigned long mask)
+{
+	return false;
+}
 static inline u64 kvm_mem_attributes_generation(struct kvm *kvm)
 {
 	return 0;
@@ -2667,6 +2675,13 @@ static inline int kvm_mem_attributes_may_write_gfn(struct kvm *kvm, gfn_t gfn)
 	return kvm_mem_attributes_may_write(attrs);
 }
 
+static inline bool kvm_range_has_rw_memory_protections(struct kvm *kvm,
+						       gfn_t start, gfn_t end)
+{
+	return kvm_range_has_any_memory_attributes(kvm, start, end,
+			KVM_MEMORY_ATTRIBUTE_NR | KVM_MEMORY_ATTRIBUTE_NW);
+}
+
 static inline bool kvm_memslots_check_gen(struct kvm *kvm, u64 slots_generation,
 					  u64 attrs_generation, struct kvm_memslots **p_slots)
 {
@@ -2682,7 +2697,6 @@ static inline bool kvm_check_gen(struct kvm *kvm, u64 slots_generation,
 	return kvm_memslots_check_gen(kvm, slots_generation, attrs_generation, &slots);
 }
 
-
 #ifdef CONFIG_KVM_GUEST_MEMFD
 int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		     gfn_t gfn, kvm_pfn_t *pfn, struct page **page,
virt/kvm/kvm_main.c +11 -4
index c00410bb62a2..2045252bd9c9 100644
@@ -3553,15 +3553,22 @@ static int __kvm_gfn_to_hva_cache_init(struct kvm *kvm,
 	 * If the requested region crosses two memslots, we still
 	 * verify that the entire region is valid here.
 	 */
-	for ( ; start_gfn <= end_gfn; start_gfn += nr_pages_avail) {
-		ghc->memslot = __gfn_to_memslot(slots, start_gfn);
-		ghc->hva = gfn_to_hva_many(ghc->memslot, start_gfn,
-					   &nr_pages_avail);
+	for (gfn_t gfn = start_gfn ; gfn <= end_gfn; gfn += nr_pages_avail) {
+		ghc->memslot = __gfn_to_memslot(slots, gfn);
+		ghc->hva = gfn_to_hva_many(ghc->memslot, gfn, &nr_pages_avail);
 		if (kvm_is_error_hva(ghc->hva))
 			return -EFAULT;
 	}
 
+	/*
+	 * RW memory attributes are incompatible with GHC. The RW protection
+	 * check has to happen after storing the generation number.
+	 */
 	ghc->attrs_generation = kvm_mem_attributes_generation(kvm);
+	if (kvm_range_has_rw_memory_protections(kvm, start_gfn, end_gfn + 1)) {
+		ghc->hva = KVM_HVA_ERR_BAD;
+		return -EFAULT;
+	}
 
 	/* Use the slow path for cross page reads and writes. */
 	if (nr_pages_needed == 1)
Signature · 1 line
2.52.0
#27 lore raw

Re: [PATCH 22/31] KVM: Take memory protections into account for accesses with cached gfn->hva

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] kvm: missing closing parenthesis for unlikely() in kvm_update_stolen_time() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] kvm: missing closing parenthesis for unlikely() in kvm_update_stolen_time() --

commit 3448cba8fa9867cc077bee10fde46930ff148f4b
Author: Nicolas Saenz Julienne <[email protected]>

KVM: Take memory protections into account for accesses with cached gfn->hva

This patch accounts for memory attributes when accessing guest memory through kvm_get/put_guest(). It tracks the memory attributes generation in gfn_to_hva_cache's data and fails to refresh the cache if restrictive memory attributes are found.

arch/loongarch/kvm/vcpu.c static void kvm_update_stolen_time(struct kvm_vcpu *vcpu) · line 168

Quoted 13 lines
arch/loongarch/kvm/vcpu.c +3 -2
index ed9e092c97ba7..1179c413b048f 100644
@@ -164,8 +163,9 @@ static void kvm_update_stolen_time(struct kvm_vcpu *vcpu)
 		return;
 
 	gpa &= KVM_STEAL_PHYS_MASK;
-	slots = kvm_memslots(vcpu->kvm);
-	if (slots->generation != ghc->generation || gpa != ghc->gpa) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {

[Severity: Low]
Does this missing closing parenthesis for the unlikely() macro cause a build failure in kvm_update_stolen_time()?

Quoted 3 lines
arch/loongarch/kvm/vcpu.c +0 -0
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;

[ ... ]

arch/loongarch/kvm/vcpu.c static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu) · line 1814

Quoted 9 lines
arch/loongarch/kvm/vcpu.c +3 -2
@@ -1810,8 +1809,9 @@ static void kvm_vcpu_set_pv_preempted(struct kvm_vcpu *vcpu)
 
 	gpa &= KVM_STEAL_PHYS_MASK;
 	ghc = &vcpu->arch.st.cache;
-	slots = kvm_memslots(vcpu->kvm);
-	if (slots->generation != ghc->generation || gpa != ghc->gpa) {
+	if (unlikely(!kvm_check_gen(vcpu->kvm, ghc->slots_generation, ghc->attrs_generation) ||
+		     gpa != ghc->gpa ||
+		     kvm_is_error_hva(ghc->hva) || !ghc->memslot) {

[Severity: Low]
Is the closing parenthesis for the unlikely() macro also missing here in kvm_vcpu_set_pv_preempted()?

Quoted 3 lines
arch/loongarch/kvm/vcpu.c +0 -0
 		if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st))) {
 			ghc->gpa = INVALID_GPA;
 			return;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=22
author maintainer
#28 lore raw

PATCH 23/31 KVM: pfncache: Fail to refresh if it contains memory protections

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Similar to how pfncache disallows caching gfns mapped within read-only memory slots, gfns marked as read-only by memory attributes must fail to initialize. Unsurprisingly, the same behaviour applies to gfns mapped as non-accessible (NR/NW), while non-executable mappings are fine.

---
 virt/kvm/pfncache.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
virt/kvm/pfncache.c +11 -0
index 46ffae69fe77..39935740136d 100644
@@ -298,7 +298,18 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned l
 		    gpc->gpa != gpa || kvm_is_error_hva(gpc->uhva)) {
 			gfn_t gfn = gpa_to_gfn(gpa);
 
+			/*
+			 * RW memory attributes are incompatible with GPC. The
+			 * RW protection check has to happen after storing the
+			 * generation number.
+			 */
 			gpc->attrs_generation = kvm_mem_attributes_generation(kvm);
+			if (kvm_range_has_rw_memory_protections(kvm, gfn, gfn + 1)) {
+				gpc->uhva = KVM_HVA_ERR_BAD;
+				ret = -EFAULT;
+				goto out;
+			}
+
 			gpc->gpa = gpa;
 			gpc->slots_generation = slots->generation;
 			gpc->memslot = __gfn_to_memslot(slots, gfn);
Signature · 1 line
2.52.0
author maintainer
#29 lore raw

PATCH 24/31 KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory attributes

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Prefetched SPTEs are always given full access. Do not prefetch GFNs that have memory protections applied.

---
 arch/x86/kvm/mmu/mmu.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/x86/kvm/mmu/mmu.c +5 -0
index 1ecb3f3fe832..3c926f95f87b 100644
@@ -3187,6 +3187,11 @@ static bool kvm_mmu_prefetch_sptes(struct kvm_vcpu *vcpu, gfn_t gfn, u64 *sptep,
 		return false;
 
 	for (i = 0; i < nr_pages; i++, gfn++, sptep++) {
+		if (kvm_get_memory_attributes(vcpu->kvm, gfn)) {
+			kvm_release_page_clean(pages[i]);
+			continue;
+		}
+
 		mmu_set_spte(vcpu, slot, sptep, access, gfn,
 			     page_to_pfn(pages[i]), NULL);
 
Signature · 1 line
2.52.0
author maintainer
#30 lore raw

PATCH 25/31 KVM: x86/mmu: Take memory protection attributes into account during faults

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

Take memory protection attributes when faulting guest memory. Prohibited memory accesses will cause a user-space -EFAULT exit just like private memory accesses. Userspace will either enable the access or bump it to the guest as some kind of exception (e.g. a VTL return).

Since the struct kvm_page_fault already has the access type in PFERR_* format, the check is done via the kvm_page_format permissions table. This means that it supports naturally all page table format variants, and it can even handle mode-based memory protection when the host uses MBEC/GMET. The only thing that needs some care is to build the restricted ACC_* mask with the root page's own access mask as a base (and not ACC_ALL). Otherwise, supervisor mode execution would be handled incorrectly on AMD processors with GMET.

To avoid spamming the trace buffer too much, the new trace event only kicks in if memory protection attributes are present for the faulted gfn.

---
 arch/x86/kvm/mmu/mmu.c          | 49 +++++++++++++++++++++++++++++++++
 arch/x86/kvm/mmu/mmu_internal.h | 19 +++++++++++++
 arch/x86/kvm/mmu/mmutrace.h     | 36 ++++++++++++++++++++++++
 arch/x86/kvm/mmu/paging_tmpl.h  |  2 +-
 arch/x86/kvm/mmu/spte.h         | 11 ++------
 5 files changed, 107 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +49 -0
index 3c926f95f87b..6bbecbae2f8c 100644
@@ -4647,6 +4647,50 @@ static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu,
 	return RET_PF_CONTINUE;
 }
 
+static inline unsigned kvm_get_gfn_protections(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm *kvm = vcpu->kvm;
+	unsigned int access = vcpu->arch.mmu->root_role.access;
+	unsigned long attrs = kvm_get_memory_attributes(kvm, gfn);
+	if (!attrs)
+		return access;
+
+	WARN_ON_ONCE(!kvm_mem_attributes_valid(kvm, attrs));
+
+	if (!kvm_mem_attributes_may_read(attrs))
+		access &= ~ACC_READ_MASK;
+	if (!kvm_mem_attributes_may_write(attrs))
+		access &= ~ACC_WRITE_MASK;
+	if (!kvm_mem_attributes_may_exec(attrs)) {
+		access &= ~ACC_EXEC_MASK;
+		if (shadow_xu_mask)
+			access &= ~ACC_USER_EXEC_MASK;
+	}
+
+	return access;
+}
+
+static int kvm_faultin_memory_protections(struct kvm_vcpu *vcpu,
+					  struct kvm_page_fault *fault)
+{
+	unsigned access;
+
+	/* Memory attributes don't apply to MMIO regions */
+	if (unlikely(!fault->slot))
+		return RET_PF_CONTINUE;
+
+	access = kvm_get_gfn_protections(vcpu, fault->gfn);
+	if (access == ACC_ALL)
+		return RET_PF_CONTINUE;
+
+	trace_kvm_faultin_memory_protections(vcpu, fault, access);
+	if (__permission_fault(vcpu->arch.mmu, access, fault))
+		return -EFAULT;
+
+	fault->host_access &= access;
+	return RET_PF_CONTINUE;
+}
+
 static int __kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 				 struct kvm_page_fault *fault)
 {
@@ -4727,6 +4771,11 @@ static int kvm_mmu_faultin_pfn(struct kvm_vcpu *vcpu,
 	if (unlikely(!slot))
 		return kvm_handle_noslot_fault(vcpu, fault, access);
 
+	if (kvm_faultin_memory_protections(vcpu, fault)) {
+		kvm_mmu_prepare_memory_fault_exit(vcpu, fault);
+		return -EFAULT;
+	}
+
 	/*
 	 * Retry the page fault if the gfn hit a memslot that is being deleted
 	 * or moved.  This ensures any existing SPTEs for the old memslot will
arch/x86/kvm/mmu/mmu_internal.h +19 -0
index 00215b9f309f..0bad60c5aab6 100644
@@ -290,6 +290,25 @@ struct kvm_page_fault {
 	bool write_fault_to_shadow_pgtable;
 };
 
+/*
+ * Returns true if the access indicated by @fault is forbidden by the existing
+ * SPTE protections.
+ */
+static inline bool __permission_fault(struct kvm_mmu *mmu, unsigned access,
+				      struct kvm_page_fault *fault)
+{
+	unsigned pfec;
+
+	/*
+	 * RSVD is handled elsewhere, and is used for SMAP in the context
+	 * of accessing fmt.permissions[].  SPTEs never use PK or SS, as
+	 * they are not supported for shadow paging and irrelevant for TDP.
+	 */
+	pfec = fault->error_code & (
+		PFERR_WRITE_MASK | PFERR_USER_MASK | PFERR_FETCH_MASK);
+	return (mmu->fmt.permissions[pfec >> 1] >> access) & 1;
+}
+
 /*
  * Return values of handle_mmio_page_fault(), mmu.page_fault(), fast_page_fault(),
  * and of course kvm_mmu_do_page_fault().
arch/x86/kvm/mmu/mmutrace.h +36 -0
index 8354d9f39777..744f111644cb 100644
@@ -8,6 +8,13 @@
 #undef TRACE_SYSTEM
 #define TRACE_SYSTEM kvmmmu
 
+#ifdef CREATE_TRACE_POINTS
+#define tracing_kvm_rip_read(vcpu) ({					\
+	typeof(vcpu) __vcpu = vcpu;					\
+	__vcpu->arch.guest_state_protected ? 0 : kvm_rip_read(__vcpu);	\
+	})
+#endif
+
 #define KVM_MMU_PAGE_FIELDS		\
 	__field(__u8, mmu_valid_gen)	\
 	__field(__u64, gfn)		\
@@ -447,6 +454,35 @@ TRACE_EVENT(
 		  __entry->gfn, __entry->spte, __entry->level, __entry->errno)
 );
 
+TRACE_EVENT(kvm_faultin_memory_protections,
+	TP_PROTO(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault,
+		 unsigned access),
+	TP_ARGS(vcpu, fault, access),
+
+	TP_STRUCT__entry(
+		__field(unsigned int, vcpu_id)
+		__field(unsigned long, guest_rip)
+		__field(u64, fault_address)
+		__field(bool, write)
+		__field(bool, exec)
+		__field(unsigned, access)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu_id = vcpu->vcpu_id;
+		__entry->guest_rip = tracing_kvm_rip_read(vcpu);
+		__entry->fault_address = fault->gfn;
+		__entry->write = fault->write;
+		__entry->exec = fault->exec;
+		__entry->access = access;
+	),
+
+	TP_printk("vcpu %d rip 0x%lx gfn 0x%016llx access %s protections 0x%x",
+		  __entry->vcpu_id, __entry->guest_rip, __entry->fault_address,
+		  __entry->exec ? "X" : (__entry->write ? "W" : "R"),
+		  __entry->access)
+);
+
 #endif /* _TRACE_KVMMMU_H */
 
 #undef TRACE_INCLUDE_PATH
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
index e6ec14165f40..b031123fcb36 100644
@@ -992,7 +992,7 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int
 
 	sptep = &sp->spt[i];
 	spte = *sptep;
-	host_access = ACC_ALL;
+	host_access = kvm_get_gfn_protections(vcpu, gfn);
 	if (!(spte & shadow_host_writable_mask))
 		host_access &= ~ACC_WRITE_MASK;
 	slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
arch/x86/kvm/mmu/spte.h +2 -9
index 589f3954633e..14f322944671 100644
@@ -491,7 +491,7 @@ static inline bool is_mmu_writable_spte(u64 spte)
 static inline bool spte_permission_fault(struct kvm_mmu *mmu, u64 spte,
 					 struct kvm_page_fault *fault)
 {
-	unsigned pfec, pte_access;
+	unsigned pte_access;
 
 	if (!is_shadow_present_pte(spte))
 		return true;
@@ -511,14 +511,7 @@ static inline bool spte_permission_fault(struct kvm_mmu *mmu, u64 spte,
 		pte_access |= spte & shadow_xu_mask ? ACC_USER_EXEC_MASK : 0;
 	}
 
-	/*
-	 * RSVD is handled elsewhere, and is used for SMAP in the context
-	 * of accessing fmt.permissions[].  SPTEs never use PK or SS, as
-	 * they are not supported for shadow paging and irrelevant for TDP.
-	 */
-	pfec = fault->error_code & (
-		PFERR_WRITE_MASK | PFERR_USER_MASK | PFERR_FETCH_MASK);
-	return (mmu->fmt.permissions[pfec >> 1] >> pte_access) & 1;
+	return __permission_fault(mmu, pte_access, fault);
 }
 
 /*
Signature · 1 line
2.52.0
#31 lore raw

Re: [PATCH 25/31] KVM: x86/mmu: Take memory protection attributes into account during faults

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] kvm: trace buffer spam on AMD with GMET enabled --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] kvm: trace buffer spam on AMD with GMET enabled
--

commit e48ad04487bab345ee62e42a2457d15e6edd7180
Author: Nicolas Saenz Julienne <[email protected]>
Subject: KVM: x86/mmu: Take memory protection attributes into account during faults

This commit adds support for memory protection attributes during guest page faults, injecting a user-space -EFAULT exit for prohibited accesses. It also adds a trace event that is intended to trigger only when memory protection attributes restrict access.

Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +0 -0
index 3c926f95f87be..6bbecbae2f8c4 100644

[ ... ]

Quoted 8 lines
arch/x86/kvm/mmu/mmu.c +8 -0
+static inline unsigned kvm_get_gfn_protections(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm *kvm = vcpu->kvm;
+	unsigned int access = vcpu->arch.mmu->root_role.access;
+	unsigned long attrs = kvm_get_memory_attributes(kvm, gfn);
+	if (!attrs)
+		return access;
+

[ ... ]

Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +4 -0
+static int kvm_faultin_memory_protections(struct kvm_vcpu *vcpu,
+					  struct kvm_page_fault *fault)
+{
+	unsigned access;

arch/x86/kvm/mmu/mmu.c static int kvm_mmu_faultin_pfn_gmem(struct kvm_vcpu *vcpu, · line 4686

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10 -0
+
+	/* Memory attributes don't apply to MMIO regions */
+	if (unlikely(!fault->slot))
+		return RET_PF_CONTINUE;
+
+	access = kvm_get_gfn_protections(vcpu, fault->gfn);
+	if (access == ACC_ALL)
+		return RET_PF_CONTINUE;
+
+	trace_kvm_faultin_memory_protections(vcpu, fault, access);

[Severity: Medium]
Will this check effectively prevent trace buffer spam on AMD processors with Guest Mode Execute Trap (GMET) enabled?

When GMET is enabled, KVM sets the root_role.access for the TDP MMU to ACC_ALL & ~ACC_USER_MASK. Since kvm_get_gfn_protections() initializes the access mask using vcpu->arch.mmu->root_role.access, it will return the restricted mask even if no memory attributes are present.

If access is restricted by GMET, the condition checking if access == ACC_ALL evaluates to false, which causes the trace event to trigger unconditionally on every page fault. Could this lead to noticeable performance regressions during tracing on these systems?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=25
author maintainer
#32 lore raw

PATCH 26/31 KVM: x86/mmu: Issue memory fault exit if walk failed due to memory attribute

In reply to Paolo Bonzini

From: Nicolas Saenz Julienne <[email protected]>

During execution, page table entries are subject to memory protection simply because their GPA is obtained with an EPT walk. During emulation, the checks need to be done by hand and result in an -EFAULT userspace exit.

However, if a guest PTE is read-only, do not go all the way out to userspace and just stomp out the write. This is consistent with the way KVM handles read-only memory slots during emulation, though admittedly inconsistent with everything else: NPT for example always wants nested page table entries for the guest page tables to be writable, and will fault.

---
 arch/x86/kvm/mmu/paging_tmpl.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/x86/kvm/mmu/paging_tmpl.h +17 -0
index b031123fcb36..cd055bfb1eb6 100644
@@ -89,6 +89,7 @@ struct guest_walker {
 	unsigned int pt_access[PT_MAX_FULL_LEVELS];
 	unsigned int pte_access;
 	gfn_t gfn;
+	bool memory_attributes_fault;
 	struct x86_exception fault;
 };
 
@@ -342,6 +343,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 
 	trace_kvm_mmu_pagetable_walk(addr, access);
 retry_walk:
+	walker->memory_attributes_fault = false;
 	walker->level = w->cpu_role.base.level;
 	pte           = kvm_mmu_get_guest_pgd(vcpu, w);
 	have_ad       = PT_HAVE_ACCESSED_DIRTY(w);
@@ -411,6 +413,15 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,
 		if (unlikely(kvm_is_error_hva(host_addr)))
 			goto error;
 
+		if (!kvm_mem_attributes_may_read_gfn(vcpu->kvm, gpa_to_gfn(real_gpa))) {
+			walker->memory_attributes_fault = true;
+			walker->gfn = gpa_to_gfn(real_gpa);
+			goto error;
+		}
+
+		if (!kvm_mem_attributes_may_write_gfn(vcpu->kvm, gpa_to_gfn(real_gpa)))
+			walker->pte_writable[walker->level - 1] = false;
+
 		ptep_user = (pt_element_t __user *)((void *)host_addr + offset);
 		if (unlikely(get_user(pte, ptep_user)))
 			goto error;
@@ -820,6 +831,12 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	 * The page is not mapped by the guest.  Let the guest handle it.
 	 */
 	if (!r) {
+		if (walker.memory_attributes_fault) {
+			kvm_prepare_memory_fault_exit(vcpu, walker.gfn << PAGE_SHIFT,
+						      PAGE_SIZE, false, false, false);
+			return -EFAULT;
+		}
+
 		if (!fault->prefetch)
 			__kvm_inject_emulated_page_fault(vcpu, &walker.fault, true);
 
Signature · 1 line
2.52.0
#33 lore raw

Re: [PATCH 26/31] KVM: x86/mmu: Issue memory fault exit if walk failed due to memory attribute

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] kvm: x86/mmu: lost memory attribute fault on GVA to GPA translation during emulation --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] kvm: x86/mmu: lost memory attribute fault on GVA to GPA translation during emulation --

commit a5b7ea3b514c528d86717bda8577133bab95a9d8
Author: Nicolas Saenz Julienne <[email protected]>

KVM: x86/mmu: Issue memory fault exit if walk failed due to memory attribute

This commit modifies the guest page table walk to issue a user-space memory fault exit if it fails due to memory attributes, instead of injecting a page fault. It introduces the memory_attributes_fault flag to the guest_walker structure.

Quoted 4 lines
arch/x86/kvm/mmu/paging_tmpl.h +0 -0
index b031123fcb369..cd055bfb1eb6b 100644

[ ... ]

arch/x86/kvm/mmu/paging_tmpl.h static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault · line 841

Quoted 12 lines
arch/x86/kvm/mmu/paging_tmpl.h +6 -0
@@ -820,6 +831,12 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault
 	 * The page is not mapped by the guest.  Let the guest handle it.
 	 */
 	if (!r) {
+		if (walker.memory_attributes_fault) {
+			kvm_prepare_memory_fault_exit(vcpu, walker.gfn << PAGE_SHIFT,
+						      PAGE_SIZE, false, false, false);
+			return -EFAULT;
+		}
+
 		if (!fault->prefetch)
 			__kvm_inject_emulated_page_fault(vcpu, &walker.fault, true);

[Severity: High]
Does FNAME(gva_to_gpa)() also need to handle the new memory_attributes_fault flag?

FNAME(gva_to_gpa)() uses a local walker to translate guest virtual addresses by calling FNAME(walk_addr_generic)():

static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w,
gpa_t addr, u64 access,
struct x86_exception *exception)

{
	struct guest_walker walker;
	gpa_t gpa = INVALID_GPA;
	int r;

[ ... ]

	r = FNAME(walk_addr_generic)(&walker, vcpu, w, addr, access);

	if (r) {
		gpa = gfn_to_gpa(walker.gfn);
		gpa |= addr & ~PAGE_MASK;
	} else if (exception)
		*exception = walker.fault;

	return gpa;
}

If FNAME(walk_addr_generic)() detects a memory attribute fault when reading guest page tables, it returns 0 with walker.memory_attributes_fault set. However, FNAME(gva_to_gpa)() appears to ignore the new flag and only copies walker.fault to the output exception.

Can this result in the instruction emulator improperly injecting a spurious page fault into the guest instead of exiting to userspace with KVM_EXIT_MEMORY_FAULT when guest page tables reside in memory restricted by KVM memory attributes?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=26
author maintainer
#34 lore raw

PATCH 27/31 KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock

In reply to Paolo Bonzini

Ensure that mmu_invalidate_in_progress is not cleared until kvm_arch_post_set_memory_attributes() returns.

kvm_arch_post_set_memory_attributes() runs with the MMU lock taken exclusively, so a concurrent page fault would be blocked anyway, and delaying the clearing makes it possible to use cond_resched() within kvm_arch_post_set_memory_attributes().

---
 virt/kvm/kvm_main.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
virt/kvm/kvm_main.c +12 -4
index 2045252bd9c9..eb4a661cd0f7 100644
@@ -2516,7 +2516,7 @@ static bool kvm_range_memory_attributes_need_sync(struct kvm *kvm,
 	return kvm_range_has_any_memory_attributes(kvm, start, end, mask);
 }
 
-static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
+static __always_inline bool kvm_handle_gfn_range(struct kvm *kvm,
 						 struct kvm_mmu_notifier_range *range)
 {
 	struct kvm_gfn_range gfn_range;
@@ -2566,6 +2566,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 
 	if (found_memslot)
 		KVM_MMU_UNLOCK(kvm);
+
+	return found_memslot;
 }
 
 static bool kvm_pre_set_memory_attributes(struct kvm *kvm,
@@ -2603,10 +2605,11 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		.end = end,
 		.arg.attributes = attributes,
 		.handler = kvm_arch_post_set_memory_attributes,
-		.on_lock = kvm_mmu_invalidate_end,
+		.on_lock = (void *)kvm_null_fn,
 		.may_block = true,
 	};
 	bool sync = false;
+	bool found_memslot;
 	unsigned long i;
 	void *entry;
 	int r = 0;
@@ -2637,7 +2640,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		cond_resched();
 	}
 
-	kvm_handle_gfn_range(kvm, &pre_set_range);
+	found_memslot = kvm_handle_gfn_range(kvm, &pre_set_range);
 
 	for (i = start; i < end; i++) {
 		r = xa_err(xa_store(&kvm->mem_attrs.array, i, entry,
@@ -2650,7 +2653,12 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	smp_store_release(&kvm->mem_attrs.generation,
 			  kvm->mem_attrs.generation + 1);
 
-	kvm_handle_gfn_range(kvm, &post_set_range);
+	if (found_memslot) {
+		kvm_handle_gfn_range(kvm, &post_set_range);
+		KVM_MMU_LOCK(kvm);
+		kvm_mmu_invalidate_end(kvm);
+		KVM_MMU_UNLOCK(kvm);
+	}
 
 out_unlock:
 	mutex_unlock(&kvm->slots_lock);
Signature · 1 line
2.52.0
author maintainer
#35 lore raw

PATCH 28/31 KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs

In reply to Paolo Bonzini

Note that the zapping of obsolete roots needs to happen after mmu_lock is dropped, so introduce a separate kvm_arch callback. The two map neatly to the "front" and "back" halves of kvm_mmu_zap_all_fast().

---
 arch/x86/kvm/mmu/mmu.c   | 30 ++++++++++++++++++++++++++++++
 include/linux/kvm_host.h |  1 +
 virt/kvm/kvm_main.c      |  1 +
 3 files changed, 32 insertions(+)
arch/x86/kvm/mmu/mmu.c +30 -0
index 6bbecbae2f8c..242cbe65859d 100644
@@ -8271,6 +8271,7 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 {
 	unsigned long attrs = range->arg.attributes;
 	struct kvm_memory_slot *slot = range->slot;
+	struct kvm_mmu_page *sp;
 	int level;
 
 	lockdep_assert_held_write(&kvm->mmu_lock);
@@ -8326,9 +8327,38 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 				hugepage_set_mixed(slot, gfn, level);
 		}
 	}
+
+	/*
+	 * There are special considerations when applying an memory protection
+	 * attibute against a page containing guest PTEs, because virtual
+	 * addresses that requires traversing that page need to fault.
+	 *
+	 * When using shadow paging, the PTEs that the CPU actually walks
+	 * do not have any information about attributes (unlike TDP where
+	 * the attributes become part of the TDP permission bits).  Since
+	 * this should be a rare occasion, just invalidate all the mmu roots
+	 * if shadow paging is in use and attributes do cover a guest PTE.
+	 */
+	for (gfn_t gfn = range->start; gfn < range->end; gfn++) {
+		for_each_gfn_valid_sp_with_gptes(kvm, sp, gfn) {
+			__kvm_mmu_zap_all_fast_front_half(kvm);
+			return false;
+		}
+	}
 	return false;
 }
 
+void kvm_arch_post_set_memory_attributes_unlocked(struct kvm *kvm)
+{
+	/*
+	 * This may also be called if no zap was done in
+	 * kvm_arch_post_set_memory_attributes().  It is not an issue because
+	 * __kvm_mmu_zap_all_fast_back_half() is cheap when it does not do
+	 * anything.
+	 */
+	 __kvm_mmu_zap_all_fast_back_half(kvm);
+}
+
 void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
 					    struct kvm_memory_slot *slot)
 {
include/linux/kvm_host.h +1 -0
index e9c0932f150e..33f3199a855f 100644
@@ -2624,6 +2624,7 @@ bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 					struct kvm_gfn_range *range);
 bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 					 struct kvm_gfn_range *range);
+void kvm_arch_post_set_memory_attributes_unlocked(struct kvm *kvm);
 bool kvm_mem_attributes_valid(struct kvm *kvm, unsigned long attrs);
 u64 kvm_supported_mem_attributes(struct kvm *kvm);
 
virt/kvm/kvm_main.c +1 -0
index eb4a661cd0f7..653bb85b3cdd 100644
@@ -2658,6 +2658,7 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		KVM_MMU_LOCK(kvm);
 		kvm_mmu_invalidate_end(kvm);
 		KVM_MMU_UNLOCK(kvm);
+		kvm_arch_post_set_memory_attributes_unlocked(kvm);
 	}
 
 out_unlock:
Signature · 1 line
2.52.0
#36 lore raw

Re: [PATCH 28/31] KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] kvm: soft lockup in KVM_SET_MEMORY_ATTRIBUTES due to unyielding loop --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] kvm: soft lockup in KVM_SET_MEMORY_ATTRIBUTES due to unyielding loop --

commit c7a249c1f674a4bf236247e82b5eb5accc8a2958
Author: Paolo Bonzini <[email protected]>

KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs

This commit introduces a separate architecture callback for the back half of kvm_mmu_zap_all_fast() so that obsolete roots can be zapped after the mmu_lock is dropped. It applies this by checking for guest PTEs when setting memory attributes and triggering a front-half fast zap if found.

Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +0 -0
index 6bbecbae2f8c4..242cbe65859df 100644

[ ... ]

Quoted 1 line
arch/x86/kvm/mmu/mmu.c +0 -0
@@ -8326,9 +8327,38 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,

[ ... ]

Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +9 -0
+	/*
+	 * There are special considerations when applying an memory protection
+	 * attibute against a page containing guest PTEs, because virtual
+	 * addresses that requires traversing that page need to fault.
+	 *
+	 * When using shadow paging, the PTEs that the CPU actually walks
+	 * do not have any information about attributes (unlike TDP where
+	 * the attributes become part of the TDP permission bits).  Since
+	 * this should be a rare occasion, just invalidate all the mmu roots

arch/x86/kvm/mmu/mmu.c bool kvm_arch_post_set_memory_attributes(struct kvm *kvm, · line 8349

Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +8 -0
+	 * if shadow paging is in use and attributes do cover a guest PTE.
+	 */
+	for (gfn_t gfn = range->start; gfn < range->end; gfn++) {
+		for_each_gfn_valid_sp_with_gptes(kvm, sp, gfn) {
+			__kvm_mmu_zap_all_fast_front_half(kvm);
+			return false;
+		}
+	}
 	return false;
 }

[Severity: High]
Does this loop present a denial of service vector? If userspace calls the KVM_SET_MEMORY_ATTRIBUTES ioctl with a very large memory range (for example, 1TB), this loop will execute hundreds of millions of iterations doing hash table lookups.

Because kvm_arch_post_set_memory_attributes() is called with the global kvm->mmu_lock held for write, and the loop scales linearly with the size of the update without yielding, can an unprivileged userspace process stall the host CPU indefinitely, causing RCU stalls and soft lockups?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=28
author maintainer
#37 lore raw

PATCH 29/31 KVM: x86: selftests: Introduce memory protection attributes test

In reply to Paolo Bonzini

Introduce architecture-independent tests for memory attributes, using small architecture-specific snippets to handle the execution tests.

---
 tools/include/uapi/linux/kvm.h                |   3 +
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../testing/selftests/kvm/include/kvm_util.h  |  23 +-
 .../testing/selftests/kvm/memory_attributes.c | 373 ++++++++++++++++++
 .../selftests/kvm/x86/memory_attributes.c     |  43 ++
 5 files changed, 433 insertions(+), 10 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/memory_attributes.c
 create mode 100644 tools/testing/selftests/kvm/x86/memory_attributes.c
tools/include/uapi/linux/kvm.h +3 -0
index 419011097fa8..9bd6488adc62 100644
@@ -1649,6 +1649,9 @@ struct kvm_memory_attributes {
 	__u64 flags;
 };
 
+#define KVM_MEMORY_ATTRIBUTE_NR		       (1ULL << 0)
+#define KVM_MEMORY_ATTRIBUTE_NW		       (1ULL << 1)
+#define KVM_MEMORY_ATTRIBUTE_NX		       (1ULL << 2)
 #define KVM_MEMORY_ATTRIBUTE_PRIVATE           (1ULL << 3)
 
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
tools/testing/selftests/kvm/Makefile.kvm +1 -0
index 6e2bbf743eaa..776b91c800d0 100644
@@ -167,6 +167,7 @@ TEST_GEN_PROGS_x86 += rseq_test
 TEST_GEN_PROGS_x86 += steal_time
 TEST_GEN_PROGS_x86 += system_counter_offset_test
 TEST_GEN_PROGS_x86 += pre_fault_memory_test
+TEST_GEN_PROGS_x86 += memory_attributes
 
 # Compiled outputs used by test targets
 TEST_GEN_PROGS_EXTENDED_x86 += x86/nx_huge_pages_test
tools/testing/selftests/kvm/include/kvm_util.h +13 -10
index 5a7a455b5387..996909230ac7 100644
@@ -418,24 +418,27 @@ static inline void vm_enable_cap(struct kvm_vm *vm, u32 cap, u64 arg0)
 	vm_ioctl(vm, KVM_ENABLE_CAP, &enable_cap);
 }
 
-static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
-					    u64 size, u64 attributes)
+static inline int __vm_set_memory_attributes(struct kvm_vm *vm, u64 gpa,
+					     u64 size, u64 attributes,
+					     u64 flags)
 {
 	struct kvm_memory_attributes attr = {
 		.attributes = attributes,
 		.address = gpa,
 		.size = size,
-		.flags = 0,
+		.flags = flags,
 	};
 
-	/*
-	 * KVM_SET_MEMORY_ATTRIBUTES overwrites _all_ attributes.  These flows
-	 * need significant enhancements to support multiple attributes.
-	 */
-	TEST_ASSERT(!attributes || attributes == KVM_MEMORY_ATTRIBUTE_PRIVATE,
-		    "Update me to support multiple attributes!");
+	return __vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
+}
 
-	vm_ioctl(vm, KVM_SET_MEMORY_ATTRIBUTES, &attr);
+static inline void vm_set_memory_attributes(struct kvm_vm *vm, gpa_t gpa,
+					    u64 size, u64 attributes)
+{
+	int rc;
+
+	rc = __vm_set_memory_attributes(vm, gpa, size, attributes, 0);
+	TEST_ASSERT_VM_VCPU_IOCTL(!rc, KVM_SET_MEMORY_ATTRIBUTES, rc, vm);
 }
 
 
tools/testing/selftests/kvm/memory_attributes.c new file +373 -0

373 lines not on this page · raw message

tools/testing/selftests/kvm/x86/memory_attributes.c new file +43 -0

43 lines not on this page · raw message

Signature · 1 line
2.52.0
author maintainer
#38 lore raw

PATCH 30/31 KVM: x86: selftests: Introduce memory attributes PTE test

In reply to Paolo Bonzini

Add more memory attributes tests to check that memory protection restricts accesses even when installed on a page that holds guest page table entries. Notably two cases are taken into account:
- The page is made non-accesible. In such case the next access to a virtual memory address translated by that paging structure should fault.
- The page is made read-only. In such case the next access to a virtual memory address translated by that paging structure should either fault, or succeed yet not perform any writes into the PTE (accessed and dirty bits).

---
 .../testing/selftests/kvm/memory_attributes.c |  39 ++++-
 .../selftests/kvm/x86/memory_attributes.c     | 144 +++++++++++++++++-
 2 files changed, 177 insertions(+), 6 deletions(-)
tools/testing/selftests/kvm/memory_attributes.c +35 -4
index 1e6e78f43100..c703c5586d43 100644
@@ -25,11 +25,16 @@
 #define MMIO_GPA	0x700000000
 #define MMIO_GVA	MMIO_GPA
 
+#define PT_WRITABLE_MASK	BIT_ULL(1)
+#define PT_ACCESSED_MASK	BIT_ULL(5)
+#define PTE_VADDR		0x1000000000
+
 enum {
 	TEST_OP_NOP,
 	TEST_OP_READ,
 	TEST_OP_WRITE,
 	TEST_OP_EXEC,
+	TEST_OP_INVPLG,
 	TEST_OP_EXIT,
 };
 
@@ -38,6 +43,7 @@ const char *test_op_names[] =
 	[TEST_OP_READ] = "Read",
 	[TEST_OP_WRITE] = "Write",
 	[TEST_OP_EXEC] = "Exec",
+	[TEST_OP_INVPLG] = "Invplg",
 	[TEST_OP_EXIT] = "Exit",
 };
 
@@ -45,6 +51,7 @@ struct test_data {
 	uint8_t op;
 	int stage;
 	gva_t vaddr;
+	uint64_t expected_val;
 
 	struct kvm_vcpu *vcpu;
 };
@@ -54,7 +61,8 @@ static struct test_data *test_data;
 static uint64_t arch_controlled_read(gva_t addr);
 static void arch_controlled_write(gva_t addr, uint64_t val);
 static void arch_controlled_exec(gva_t addr);
-static void arch_write_return_insn(struct kvm_vm *vm, gpa_t vaddr);
+static void arch_write_return_insn(struct kvm_vm *vm, gpa_t paddr);
+static bool arch_test_op(struct test_data *test_data);
 
 static void guest_code(void *data)
 {
@@ -62,6 +70,7 @@ static void guest_code(void *data)
 	int stage = 1;
 
 	while (true) {
+		uint64_t expected_val = READ_ONCE(test_data->expected_val);
 		gva_t vaddr = READ_ONCE(test_data->vaddr);
 
 		switch(READ_ONCE(test_data->op)) {
@@ -70,7 +79,7 @@ static void guest_code(void *data)
 			GUEST_SYNC(stage++);
 			break;
 		case TEST_OP_WRITE:
-			arch_controlled_write(vaddr, 1);
+			arch_controlled_write(vaddr, expected_val);
 			GUEST_SYNC(stage++);
 			break;
 		case TEST_OP_EXEC:
@@ -78,7 +87,10 @@ static void guest_code(void *data)
 			GUEST_SYNC(stage++);
 			break;
 		default:
-			goto exit;
+			if (!arch_test_op(test_data))
+				goto exit;
+			GUEST_SYNC(stage++);
+			break;
 		};
 	}
 
@@ -109,6 +121,21 @@ static void vcpu_run_and_inc_stage(struct kvm_vcpu *vcpu)
 	test_data->stage++;
 }
 
+static int test_page(struct kvm_vcpu *vcpu, int op, gva_t vaddr)
+{
+	int rc;
+
+	test_data->op = op;
+	test_data->vaddr = vaddr;
+
+	rc = _vcpu_run(vcpu);
+
+	if (rc >= 0)
+		test_data->stage++;
+
+	return rc < 0 ? -errno : rc;
+}
+
 static void test_page_restricted(struct kvm_vcpu *vcpu, int op,
 				 gva_t vaddr, gpa_t fault_paddr,
 				 uint64_t fault_reason)
@@ -125,7 +152,8 @@ static void test_page_restricted(struct kvm_vcpu *vcpu, int op,
 		    test_op_names[op], rc, errno);
 	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MEMORY_FAULT);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.gpa, fault_paddr);
-	TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, fault_reason);
+	if (fault_reason)
+		TEST_ASSERT_EQ(vcpu->run->memory_fault.flags, fault_reason);
 	TEST_ASSERT_EQ(vcpu->run->memory_fault.size, vm->page_size);
 }
 
@@ -366,6 +394,9 @@ int main(int argc, char *argv[])
 	test_input_validation(vm);
 	test_memory_access(vcpu, test_mem, size);
 	test_memattrs_ignore_mmio(vcpu);
+#ifdef __x86_64__
+	arch_test_memory_access_pte(vcpu, test_mem);
+#endif
 	test_finalize(vcpu);
 
 	kvm_vm_free(vm);
tools/testing/selftests/kvm/x86/memory_attributes.c +142 -2

149 lines not on this page · raw message

Signature · 1 line
2.52.0
author maintainer
#39 lore raw

PATCH 31/31 KVM: x86: selftests: Introduce memory attributes side-channel tests

In reply to Paolo Bonzini

Introduce memory attributes selftests to catch any vulnerable side-channels.

Memory attributes access restrictions are vulnerable to side-channel attacks. This means that any KVM operation initiated by the guest that requires guest memory access (which is the case for most para-virtualised interfaces) needs to consider memory attributes.

The tests confirm this requirement is upheld for a variety of use-cases, exercising various kinds of guest memory access such as kvm_read/write_guest(), gfn_to_hva_cache and gfn_to_pfn_cache.

---
 .../testing/selftests/kvm/include/kvm_util.h  |   9 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../testing/selftests/kvm/lib/x86/processor.c |   5 +
 .../testing/selftests/kvm/memory_attributes.c |  51 +++-
 .../selftests/kvm/x86/memory_attributes.c     | 232 ++++++++++++++++++
 5 files changed, 297 insertions(+), 1 deletion(-)
tools/testing/selftests/kvm/include/kvm_util.h +9 -0
index 996909230ac7..b18af4f5f615 100644
@@ -883,6 +883,15 @@ static inline int vcpu_get_stats_fd(struct kvm_vcpu *vcpu)
 	return fd;
 }
 
+static inline struct kvm_translation vcpu_translate(struct kvm_vcpu *vcpu,
+						    u64 gva)
+{
+	struct kvm_translation tr = { .linear_address = gva };
+
+	vcpu_ioctl(vcpu, KVM_TRANSLATE, &tr);
+	return tr;
+}
+
 int __kvm_has_device_attr(int dev_fd, u32 group, u64 attr);
 
 static inline void kvm_has_device_attr(int dev_fd, u32 group, u64 attr)
tools/testing/selftests/kvm/include/x86/processor.h +1 -0
index 6e6f70035508..398380157fa1 100644
@@ -1472,6 +1472,7 @@ static inline bool kvm_is_lbrv_enabled(void)
 	return !!get_kvm_amd_param_integer("lbrv");
 }
 
+u64 *vm_get_pte_level(struct kvm_vm *vm, gva_t gva, int *level);
 u64 *vm_get_pte(struct kvm_vm *vm, gva_t gva);
 
 u64 kvm_hypercall(u64 nr, u64 a0, u64 a1, u64 a2, u64 a3);
tools/testing/selftests/kvm/lib/x86/processor.c +5 -0
index d31fa81ea075..f51b4929d668 100644
@@ -400,6 +400,11 @@ u64 *tdp_get_pte(struct kvm_vm *vm, u64 l2_gpa)
 	return __vm_get_page_table_entry(vm, &vm->stage2_mmu, l2_gpa, &level);
 }
 
+u64 *vm_get_pte_level(struct kvm_vm *vm, gva_t gva, int *level)
+{
+	return __vm_get_page_table_entry(vm, &vm->mmu, gva, level);
+}
+
 u64 *vm_get_pte(struct kvm_vm *vm, gva_t gva)
 {
 	int level = PG_LEVEL_4K;
tools/testing/selftests/kvm/memory_attributes.c +50 -1

123 lines not on this page · raw message

tools/testing/selftests/kvm/x86/memory_attributes.c +232 -0

252 lines not on this page · raw message

Signature · 1 line
2.52.0
#40 lore raw

Re: [PATCH 31/31] KVM: x86: selftests: Introduce memory attributes side-channel tests

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] kselftests: race condition in pvclock test using usleep() for synchronization --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] kselftests: race condition in pvclock test using usleep() for synchronization --

commit 9a779483841fdb39484cbe06ffba510ee690c56e
Author: Paolo Bonzini <[email protected]>

KVM: x86: selftests: Introduce memory attributes side-channel tests

This commit introduces memory attributes selftests to catch vulnerable side-channels. It validates that KVM operations requiring memory access properly account for memory attributes across various use cases.

Quoted 3 lines
tools/testing/selftests/kvm/x86/memory_attributes.c +0 -0

[ ... ]

Quoted 1 line
tools/testing/selftests/kvm/x86/memory_attributes.c +0 -0
@@ -181,3 +192,224 @@ static void arch_test_memory_access_pte(struct kvm_vcpu *vcpu, gva_t vaddr)

[ ... ]

Quoted 7 lines
tools/testing/selftests/kvm/x86/memory_attributes.c +7 -0
+static void test_side_channel_pvclock(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vm *vm = vcpu->vm;
+	gva_t vaddr = vm_alloc_page(vm);
+	gpa_t paddr = addr_gva2gpa(vm, vaddr);
+	struct pvclock_vcpu_time_info *pvclock = addr_gpa2hva(vm, paddr);
+

tools/testing/selftests/kvm/x86/memory_attributes.c static void arch_test_memory_access_pte(struct kvm_vcpu *vcpu, gva_t vaddr) · line 352

Quoted 10 lines
tools/testing/selftests/kvm/x86/memory_attributes.c +10 -0
+	pvclock->version = 0;
+
+	test_data->op = TEST_OP_MONITOR_ADDRESS;
+	test_data->vaddr = vaddr + offsetof(struct pvclock_vcpu_time_info, version);
+	test_data->expected_val = pvclock->version;
+
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, paddr | 0x1);
+	pthread_create(&vcpu_thread, NULL, vcpu_worker, test_data);
+	usleep(msecs_to_usecs(1000));
+	TEST_ASSERT_EQ(pthread_tryjoin_np(vcpu_thread, NULL), 0);

[Severity: Medium]
Does this test rely on a hardcoded 1-second timeout for synchronization?

On heavily loaded CI systems or nested virtualized environments, vCPU scheduling and the pvclock update can easily exceed 1 second. This could cause pthread_tryjoin_np() to fail unexpectedly and create a false positive regression.

Similarly, when waiting to confirm the vCPU does not exit in the second part of this test, a slow system might simply not have scheduled the thread yet, leading to a false negative pass even if memory was erroneously updated.

tools/testing/selftests/kvm/x86/memory_attributes.c static void arch_test_memory_access_pte(struct kvm_vcpu *vcpu, gva_t vaddr) · line 364

Quoted 12 lines
tools/testing/selftests/kvm/x86/memory_attributes.c +12 -0
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, 0);
+
+	test_data->expected_val = pvclock->version;
+	vm_set_memory_attributes(vm, paddr, vm->page_size, KVM_MEMORY_ATTRIBUTE_NW);
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, paddr | 0x1);
+	pthread_create(&vcpu_thread, NULL, vcpu_worker, test_data);
+	usleep(msecs_to_usecs(1000));
+	TEST_ASSERT_EQ(pthread_tryjoin_np(vcpu_thread, NULL), EBUSY);
+
+	/* Force the 'monitor_address' guest operation to finish */
+	test_data->op = TEST_OP_NOP;
+	TEST_ASSERT_EQ(pthread_join(vcpu_thread, NULL), 0);

[Severity: Medium]
Is a WRITE_ONCE() missing when assigning test_data->op?

The test_data->op variable is polled concurrently by the guest vCPU thread via READ_ONCE() in guest_code(), so using a plain assignment here violates standard lockless concurrency patterns.

Quoted 3 lines
tools/testing/selftests/kvm/x86/memory_attributes.c +3 -0
+	vm_set_memory_attributes(vm, paddr, vm->page_size, 0);
+	vcpu_set_msr(vcpu, MSR_KVM_SYSTEM_TIME_NEW, 0);
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=31
author maintainer
#41 lore raw
In reply to Paolo Bonzini
Quoted 17 lines · Paolo Bonzini

[cover letter copied from v2 - sorry for the very quick turnaround but it's pointless to ask for reviews with the issues reported by sashiko]

This series introduces a mechanism to let userspace block read, write or execute access to individual GFNs via KVM's memory attribute mechanism, and have them reported via KVM_EXIT_MEMORY_FAULT. It is mostly the work of Nicolas Saenz Julienne, with my working consisting in reorganization, code cleanup, and using the recently revamped MMU code (ACC_* masks and kvm_page_format).

KVM needs to check the attributes anytime KVM takes GPAs as input for any action initiated by the guest; if the memory attributes are incompatible with such action, it should be stopped. This means that there are quite a lot of cases to handle. While some families of functions can be handled in one step, there are simply many places that do memory access.

Quoted 4 lines

Along the way, the patches fix some issues in the memory attributes code, that surfaced due to having more than one attribute.

Paolo

The remaining AI review issues (https://sashiko.dev/#/patchset/20260918135030.171564-2-pbonzini%40redhat.com) are simple or intentional with one exception:

- patch 21 - missing xas.xa_node == XAS_BOUNDS check, easy enough

- patch 26 - missing EFAULT returns from the emulator, need to look at it and write tests

- patch 28 - i think it should be fine, but it can also be fixed with cond_resched() if needed

- patch 31 - WRITE_ONCE can be used for completeness

If desired, patches 1 and 5 can be applied to stable and/or 7.2-rc.

Paolo

Quoted 101 lines

v2->v3 is just a bunch of sashiko fixes:
- new patch "KVM: x86/hyperv: do not overwrite hc->ingpa for slow SIGNAL_EVENT hypercall"
- rewritten "KVM: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING" to handle cross-patch accesses
- rewrite choice of KVM_FILTER_SHARED/KVM_FILTER_PRIVATE prior to setting memory attributes; even though the previous version worked, it relied on not having any attribute at all for has_private_mem VMs
- fix WARN_ON_ONCE/WARN_ONCE confusion
- fix loongarch compilation
- new (not really satisfactory_ patch "KVM: Take memory protections into account for __kvm_vcpu_map"
- fix incorrect NOT in "KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory attributes"
- do not use kvm_mmu_prepare_memory_fault_exit() for attribute exits on PTEs
- new patch "KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock"
- testcase fixes

Anish Moorthy (1):
KVM: Define and communicate KVM_EXIT_MEMORY_FAULT RWX flags to
userspace

Nicolas Saenz Julienne (12):
KVM: selftests: Take into account mixed memory fault flags
KVM: x86/mmu: Init memslot hugepage information for non-private_mem
VMs too
KVM: Introduce NR/NW/NX memory attributes
KVM: Include memory protections in result of gfn->hva conversion
KVM: Take memory protections into account for memory read/write/fetch
KVM: Encapsulate memattrs array into anonymous struct
KVM: Introduce a generation number for memory attributes
KVM: Take memory protections into account for accesses with cached

     gfn->hva

KVM: pfncache: Fail to refresh if it contains memory protections
KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory
attributes
KVM: x86/mmu: Take memory protection attributes into account during
faults
KVM: x86/mmu: Issue memory fault exit if walk failed due to memory
attribute

Paolo Bonzini (18):
KVM: x86/hyperv: do not overwrite hc->ingpa for slow SIGNAL_EVENT
hypercall
KVM: selftests: Test address translation for Hyper-V direct L2
hypercalls
KVM: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING
KVM: x86: Introduce memory fault on invalid hypercalls reads/writes
KVM: selftests: test hypercall memory fault exits
KVM: x86/mmu: intersect writability from __kvm_faultin_pfn with

     fault->map_writable

KVM: x86/mmu: Extend map_writable to a full ACC_* mask
KVM: pass kvm == NULL case to kvm_arch_has_private_mem
KVM: adjust for presence of more than one attribute
KVM: Introduce kvm_fetch_guest_page() and use it for x86
KVM: Take memory protections into account for __kvm_vcpu_map
KVM: loongarch: do full validity check on the gfn-to-hva cache
KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()
KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock
KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs
KVM: x86: selftests: Introduce memory protection attributes test
KVM: x86: selftests: Introduce memory attributes PTE test
KVM: x86: selftests: Introduce memory attributes side-channel tests

  Documentation/virt/kvm/api.rst                |  38 +-
  arch/loongarch/kvm/vcpu.c                     |  12 +-
  arch/x86/include/asm/kvm_host.h               |   4 +-
  arch/x86/kvm/Kconfig                          |   4 +-
  arch/x86/kvm/hyperv.c                         | 172 +++++--
  arch/x86/kvm/mmu/mmu.c                        | 183 +++++--
  arch/x86/kvm/mmu/mmu_internal.h               |  21 +-
  arch/x86/kvm/mmu/mmutrace.h                   |  36 ++
  arch/x86/kvm/mmu/paging_tmpl.h                |  25 +-
  arch/x86/kvm/mmu/spte.c                       |  12 +-
  arch/x86/kvm/mmu/spte.h                       |  13 +-
  arch/x86/kvm/mmu/tdp_mmu.c                    |   2 +-
  arch/x86/kvm/x86.c                            |  66 ++-
  include/linux/kvm_host.h                      | 128 ++++-
  include/linux/kvm_types.h                     |   6 +-
  include/trace/events/kvm.h                    |  14 +-
  include/uapi/linux/kvm.h                      |   7 +
  tools/include/uapi/linux/kvm.h                |   3 +
  tools/testing/selftests/kvm/Makefile.kvm      |   2 +
  .../testing/selftests/kvm/include/kvm_util.h  |  32 +-
  .../selftests/kvm/include/x86/processor.h     |   1 +
  .../testing/selftests/kvm/lib/x86/processor.c |   5 +
  .../testing/selftests/kvm/memory_attributes.c | 453 ++++++++++++++++++
  tools/testing/selftests/kvm/x86/hcall_fault.c | 246 ++++++++++
  .../testing/selftests/kvm/x86/hyperv_evmcs.c  |  16 +-
  .../selftests/kvm/x86/hyperv_svm_test.c       |  15 +-
  .../selftests/kvm/x86/memory_attributes.c     | 415 ++++++++++++++++
  .../kvm/x86/private_mem_kvm_exits_test.c      |   6 +-
  virt/kvm/kvm_main.c                           | 271 +++++++++--
  virt/kvm/pfncache.c                           |  30 +-
  30 files changed, 2027 insertions(+), 211 deletions(-)

create mode 100644 tools/testing/selftests/kvm/memory_attributes.c create mode 100644 tools/testing/selftests/kvm/x86/hcall_fault.c create mode 100644 tools/testing/selftests/kvm/x86/memory_attributes.c

maintainer
#42 lore raw
In reply to Paolo Bonzini
Quoted 17 lines · Paolo Bonzini

[cover letter copied from v2 - sorry for the very quick turnaround but it's pointless to ask for reviews with the issues reported by sashiko]

This series introduces a mechanism to let userspace block read, write or execute access to individual GFNs via KVM's memory attribute mechanism, and have them reported via KVM_EXIT_MEMORY_FAULT. It is mostly the work of Nicolas Saenz Julienne, with my working consisting in reorganization, code cleanup, and using the recently revamped MMU code (ACC_* masks and kvm_page_format).

KVM needs to check the attributes anytime KVM takes GPAs as input for any action initiated by the guest; if the memory attributes are incompatible with such action, it should be stopped. This means that there are quite a lot of cases to handle. While some families of functions can be handled in one step, there are simply many places that do memory access.

Quoted 8 lines

Along the way, the patches fix some issues in the memory attributes code, that surfaced due to having more than one attribute.

Paolo

v2->v3 is just a bunch of sashiko fixes:
- new patch "KVM: x86/hyperv: do not overwrite hc->ingpa for slow SIGNAL_EVENT hypercall"

A nitpick: in a few places in this series you use 'slow Hyper-V hypercall' term but TLFS never calls them that. It's kind of my fault I guess as in aee738236dca I used in in the comment I've added (although, in quotes to emphasize I'm making the term up). In TLFS, hypercalls are either regular (using memory locations for in/out params) or 'fast' (using registers, including XMM). I think we need to decide if we want to adopt the 'slow' term and use it ever after of if we want to fall back to a more conventional "non-fast"/"regular"/"" name.

Quoted 96 lines

- rewritten "KVM: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING" to handle cross-patch accesses
- rewrite choice of KVM_FILTER_SHARED/KVM_FILTER_PRIVATE prior to setting memory attributes; even though the previous version worked, it relied on not having any attribute at all for has_private_mem VMs
- fix WARN_ON_ONCE/WARN_ONCE confusion
- fix loongarch compilation
- new (not really satisfactory_ patch "KVM: Take memory protections into account for __kvm_vcpu_map"
- fix incorrect NOT in "KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory attributes"
- do not use kvm_mmu_prepare_memory_fault_exit() for attribute exits on PTEs
- new patch "KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock"
- testcase fixes

Anish Moorthy (1):
KVM: Define and communicate KVM_EXIT_MEMORY_FAULT RWX flags to
userspace

Nicolas Saenz Julienne (12):
KVM: selftests: Take into account mixed memory fault flags
KVM: x86/mmu: Init memslot hugepage information for non-private_mem
VMs too
KVM: Introduce NR/NW/NX memory attributes
KVM: Include memory protections in result of gfn->hva conversion
KVM: Take memory protections into account for memory read/write/fetch
KVM: Encapsulate memattrs array into anonymous struct
KVM: Introduce a generation number for memory attributes
KVM: Take memory protections into account for accesses with cached

    gfn->hva

KVM: pfncache: Fail to refresh if it contains memory protections
KVM: x86/mmu: Do not prefetch sptes on gfns backed by memory
attributes
KVM: x86/mmu: Take memory protection attributes into account during
faults
KVM: x86/mmu: Issue memory fault exit if walk failed due to memory
attribute

Paolo Bonzini (18):
KVM: x86/hyperv: do not overwrite hc->ingpa for slow SIGNAL_EVENT
hypercall
KVM: selftests: Test address translation for Hyper-V direct L2
hypercalls
KVM: apply nGPA->GPA translation to KVM_HC_CLOCK_PAIRING
KVM: x86: Introduce memory fault on invalid hypercalls reads/writes
KVM: selftests: test hypercall memory fault exits
KVM: x86/mmu: intersect writability from __kvm_faultin_pfn with

    fault->map_writable

KVM: x86/mmu: Extend map_writable to a full ACC_* mask
KVM: pass kvm == NULL case to kvm_arch_has_private_mem
KVM: adjust for presence of more than one attribute
KVM: Introduce kvm_fetch_guest_page() and use it for x86
KVM: Take memory protections into account for __kvm_vcpu_map
KVM: loongarch: do full validity check on the gfn-to-hva cache
KVM: Introduce kvm_check_gen()/kvm_memslots_check_gen()
KVM: let kvm_arch_post_set_memory_attributes drop mmu_lock
KVM: x86/mmu: Obsolete all roots if memattr contains gPTEs
KVM: x86: selftests: Introduce memory protection attributes test
KVM: x86: selftests: Introduce memory attributes PTE test
KVM: x86: selftests: Introduce memory attributes side-channel tests

 Documentation/virt/kvm/api.rst                |  38 +-
 arch/loongarch/kvm/vcpu.c                     |  12 +-
 arch/x86/include/asm/kvm_host.h               |   4 +-
 arch/x86/kvm/Kconfig                          |   4 +-
 arch/x86/kvm/hyperv.c                         | 172 +++++--
 arch/x86/kvm/mmu/mmu.c                        | 183 +++++--
 arch/x86/kvm/mmu/mmu_internal.h               |  21 +-
 arch/x86/kvm/mmu/mmutrace.h                   |  36 ++
 arch/x86/kvm/mmu/paging_tmpl.h                |  25 +-
 arch/x86/kvm/mmu/spte.c                       |  12 +-
 arch/x86/kvm/mmu/spte.h                       |  13 +-
 arch/x86/kvm/mmu/tdp_mmu.c                    |   2 +-
 arch/x86/kvm/x86.c                            |  66 ++-
 include/linux/kvm_host.h                      | 128 ++++-
 include/linux/kvm_types.h                     |   6 +-
 include/trace/events/kvm.h                    |  14 +-
 include/uapi/linux/kvm.h                      |   7 +
 tools/include/uapi/linux/kvm.h                |   3 +
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../testing/selftests/kvm/include/kvm_util.h  |  32 +-
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../testing/selftests/kvm/lib/x86/processor.c |   5 +
 .../testing/selftests/kvm/memory_attributes.c | 453 ++++++++++++++++++
 tools/testing/selftests/kvm/x86/hcall_fault.c | 246 ++++++++++
 .../testing/selftests/kvm/x86/hyperv_evmcs.c  |  16 +-
 .../selftests/kvm/x86/hyperv_svm_test.c       |  15 +-
 .../selftests/kvm/x86/memory_attributes.c     | 415 ++++++++++++++++
 .../kvm/x86/private_mem_kvm_exits_test.c      |   6 +-
 virt/kvm/kvm_main.c                           | 271 +++++++++--
 virt/kvm/pfncache.c                           |  30 +-
 30 files changed, 2027 insertions(+), 211 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/memory_attributes.c
 create mode 100644 tools/testing/selftests/kvm/x86/hcall_fault.c
 create mode 100644 tools/testing/selftests/kvm/x86/memory_attributes.c
Signature · 1 line
Vitaly