KVM: guest memory: Misc enhacnement
v3 cover KVM: guest memory: Misc enhacnement
From: Isaku Yamahata <[email protected]>
Hello. I've updated the patch series based on the feedback. Here are the discussion points.
- 06/11 KVM: x86: Introduce PFERR_GUEST_ENC_MASK to indicate fault is private Michael has his own opinion on how to indicate private fault.
- 09/11 KVM: Add new members to struct kvm_gfn_range to operate on
- 10/11 KVM: x86: Add gmem hook for initializing private memory SNP needs the callback. TDX won't use this one.
- 11/11 KVM: x86: Add gmem hook for invalidating private memory SNP needs the callback. TDX doesn't use this one at the moment, but would use it.
- VM type:
I didn't rename KVM_X86_PROTECTED_VM to KVM_X86_SW_PROTECTED_VM in this patch series. It's easy to rename it if desired.
Thanks,
This is an RFC patch series based on KVM gmem [1] and [2] for the common use of TDX and SEV-SNP.
[1] KVM gmem patches
https://github.com/sean-jc/linux/tree/x86/kvm_gmem_solo
[2] Add AMD Secure Nested Paging (SEV-SNP) Hypervisor Support
https://lore.kernel.org/lkml/[email protected]/
---
v3:
- Imported common patches from Michael Roth which can be useful for both SNP and TDX. And reorder patches.
- Update struct kvm_gfn_range to drop flag, and add add only_private, and only_shared
- Update kvm_arch_set_memory_attributes() and added a x86 vendor callback for it.
v2:
https://lore.kernel.org/all/[email protected]/
v1:
https://lore.kernel.org/all/[email protected]/
Brijesh Singh (1):
KVM: x86: Export the kvm_zap_gfn_range() for the SNP use
Isaku Yamahata (8):
KVM: selftests: Fix test_add_overlapping_private_memory_regions()
KVM: selftests: Fix guest_memfd()
KVM: selftests: x86: typo in private_mem_conversions_test.c
KVM: x86: Add is_vm_type_supported callback
KVM: x86/mmu: Pass around full 64-bit error code for the KVM page
fault
KVM: x86: Introduce PFERR_GUEST_ENC_MASK to indicate fault is private
KVM: Fix set_mem_attr ioctl when error case
KVM: Add new members to struct kvm_gfn_range to operate on
Michael Roth (2):
KVM: x86: Add gmem hook for initializing private memory
KVM: x86: Add gmem hook for invalidating private memory
arch/x86/include/asm/kvm-x86-ops.h | 4 ++ arch/x86/include/asm/kvm_host.h | 12 +++++ arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/mmu.h | 2 - arch/x86/kvm/mmu/mmu.c | 51 ++++++++++++++----- arch/x86/kvm/mmu/mmu_internal.h | 20 ++++++-- arch/x86/kvm/mmu/mmutrace.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/svm/svm.c | 7 +++ arch/x86/kvm/vmx/vmx.c | 6 +++ arch/x86/kvm/x86.c | 16 +++++- arch/x86/kvm/x86.h | 2 + include/linux/kvm_host.h | 16 ++++-- .../testing/selftests/kvm/guest_memfd_test.c | 4 +- .../selftests/kvm/set_memory_region_test.c | 16 +++++- .../kvm/x86_64/private_mem_conversions_test.c | 2 +- virt/kvm/guest_mem.c | 50 +++++++++++++++++- virt/kvm/kvm_main.c | 24 +++++---- 18 files changed, 192 insertions(+), 45 deletions(-)
base-commit: be8abcec83c87d4e15ae04816b685fe260c4bcfd
Signature · 1 line
2.25.1
RFC PATCH v3 1/11 KVM: selftests: Fix test_add_overlapping_private_memory_regions()
From: Isaku Yamahata <[email protected]>
The last test in test_add_overlapping_private_memory_regions() doesn't use overlapping regions resulting in the failure. When the region is overlaps with the existing ones, the error code is EEXIST instead of EINVAL. Pass the overlapping region, and check if the errno is EEXIST.
---
Changes v2 -> v3:
- no change
Changes v1 -> v2:
- no change
--- .../selftests/kvm/set_memory_region_test.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/set_memory_region_test.c +14 -2
@@ -432,6 +432,7 @@ static void test_add_overlapping_private_memory_regions(void) {struct kvm_vm *vm;int memfd; +int r;pr_info ("Testing ADD of overlapping KVM_MEM_PRIVATE memory regions\n" );
@@ -453,8 +454,19 @@ static void test_add_overlapping_private_memory_regions(void)vm_set_user_memory_region2 (vm,MEM_REGION_SLOT ,KVM_MEM_PRIVATE ,MEM_REGION_GPA ,0 ,NULL ,-1 ,0 ); -test_invalid_guest_memfd (vm, memfd,MEM_REGION_SIZE , -"Overlapping guest_memfd() bindings should fail" ); + r =__vm_set_user_memory_region2 (vm,MEM_REGION_SLOT ,KVM_MEM_PRIVATE , +MEM_REGION_GPA *2 -MEM_REGION_SIZE , +MEM_REGION_SIZE *2 , +0 , memfd,0 ); +TEST_ASSERT (r ==-1 && errno ==EEXIST ,"%s" , +"Overlapping guest_memfd() bindings should fail" ); + + r =__vm_set_user_memory_region2 (vm,MEM_REGION_SLOT ,KVM_MEM_PRIVATE , +MEM_REGION_GPA *2 +MEM_REGION_SIZE , +MEM_REGION_SIZE *2 , +0 , memfd,0 ); +TEST_ASSERT (r ==-1 && errno ==EEXIST ,"%s" , +"Overlapping guest_memfd() bindings should fail" );close (memfd);kvm_vm_free (vm);
Signature · 1 line
2.25.1
RFC PATCH v3 2/11 KVM: selftests: Fix guest_memfd()
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Some test cases should succeed. Check !ret instead of ret.
---
Changes v2 -> v3:
- no change
Changes v1 -> v2:
- no change
--- tools/testing/selftests/kvm/guest_memfd_test.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/kvm/guest_memfd_test.c +2 -2
@@ -72,11 +72,11 @@ static void test_fallocate(int fd, size_t page_size, size_t total_size) ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , total_size, page_size); -TEST_ASSERT (ret,"fallocate(PUNCH_HOLE) at total_size should be fine (no-op)" ); +TEST_ASSERT (!ret,"fallocate(PUNCH_HOLE) at total_size should be fine (no-op)" ); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , total_size + page_size, page_size); -TEST_ASSERT (ret,"fallocate(PUNCH_HOLE) after total_size should be fine (no-op)" ); +TEST_ASSERT (!ret,"fallocate(PUNCH_HOLE) after total_size should be fine (no-op)" ); ret =fallocate (fd,FALLOC_FL_KEEP_SIZE |FALLOC_FL_PUNCH_HOLE , page_size, page_size -1 );
Signature · 1 line
2.25.1
RFC PATCH v3 3/11 KVM: selftests: x86: typo in private_mem_conversions_test.c
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Fix typo in the comment in private_mem_conversions_test.c.
---
Changes v2 -> v3:
- Newly added
--- .../testing/selftests/kvm/x86_64/private_mem_conversions_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/kvm/x86_64/private_mem_conversions_test.c +1 -1
@@ -227,7 +227,7 @@ static void *__test_mem_conversions(void *__vcpu)/* In all cases, the host should observe the shared data. */ memcmp_h (hva, uc.args[3 ], size); -/* For shared, write the new patter to guest memory. */ +/* For shared, write the new pattern to guest memory. */ if (uc.args[0 ] ==SYNC_SHARED )memset (hva, uc.args[4 ], size);break ;
Signature · 1 line
2.25.1
RFC PATCH v3 4/11 KVM: x86: Add is_vm_type_supported callback
In reply to [email protected]From: Isaku Yamahata <[email protected]>
For TDX, allow the backend can override the supported vm type. Add KVM_X86_TDX_VM to reserve the bit.
---
Changes v2 -> v3:
- no change
- didn't bother to rename KVM_X86_PROTECTED_VM to KVM_X86_SW_PROTECTED_VM
Changes v1 -> v2
- no change
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/uapi/asm/kvm.h | 1 + arch/x86/kvm/svm/svm.c | 7 +++++++ arch/x86/kvm/vmx/vmx.c | 6 ++++++ arch/x86/kvm/x86.c | 10 +++++++++- arch/x86/kvm/x86.h | 2 ++ 7 files changed, 27 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -20,6 +20,7 @@ KVM_X86_OP(hardware_disable) KVM_X86_OP(hardware_unsetup) KVM_X86_OP(has_emulated_msr) KVM_X86_OP(vcpu_after_set_cpuid) +KVM_X86_OP(is_vm_type_supported) KVM_X86_OP(vm_init) KVM_X86_OP_OPTIONAL(vm_destroy) KVM_X86_OP_OPTIONAL_RET0(vcpu_precreate)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1543,6 +1543,7 @@ struct kvm_x86_ops {bool (*has_emulated_msr)(struct kvm *kvm,u32 index);void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu); +bool (*is_vm_type_supported)(unsigned long vm_type);unsigned int vm_size;int (*vm_init)(struct kvm *kvm);void (*vm_destroy)(struct kvm *kvm);
arch/x86/include/uapi/asm/kvm.h +1 -0
@@ -561,5 +561,6 @@ struct kvm_pmu_event_filter {#define KVM_X86_DEFAULT_VM 0#define KVM_X86_PROTECTED_VM 1 +#define KVM_X86_TDX_VM 2#endif /* _ASM_X86_KVM_H */
arch/x86/kvm/svm/svm.c +7 -0
@@ -4756,6 +4756,12 @@ static void svm_vm_destroy(struct kvm *kvm)sev_vm_destroy (kvm); } +static bool svm_is_vm_type_supported (unsigned long type) +{ +/* FIXME: Check if CPU is capable of SEV. */ +return __kvm_is_vm_type_supported (type); +} +static int svm_vm_init (struct kvm *kvm) {if (!pause_filter_count || !pause_filter_thresh)
@@ -4784,6 +4790,7 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .vcpu_free = svm_vcpu_free, .vcpu_reset = svm_vcpu_reset, + .is_vm_type_supported = svm_is_vm_type_supported, .vm_size = sizeof(struct kvm_svm), .vm_init = svm_vm_init, .vm_destroy = svm_vm_destroy,
arch/x86/kvm/vmx/vmx.c +6 -0
@@ -7469,6 +7469,11 @@ static int vmx_vcpu_create(struct kvm_vcpu *vcpu)return err; } +static bool vmx_is_vm_type_supported (unsigned long type) +{ +return __kvm_is_vm_type_supported (type); +} +#define L1TF_MSG_SMT "L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n"#define L1TF_MSG_L1D "L1TF CPU bug present and virtualization mitigation disabled, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/l1tf.html for details.\n"
@@ -8138,6 +8143,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .hardware_disable = vmx_hardware_disable, .has_emulated_msr = vmx_has_emulated_msr, + .is_vm_type_supported = vmx_is_vm_type_supported, .vm_size = sizeof(struct kvm_vmx), .vm_init = vmx_vm_init, .vm_destroy = vmx_vm_destroy,
arch/x86/kvm/x86.c +9 -1
@@ -4418,12 +4418,18 @@ static int kvm_ioctl_get_supported_hv_cpuid(struct kvm_vcpu *vcpu,return 0 ; } -static bool kvm_is_vm_type_supported (unsigned long type) +bool __kvm_is_vm_type_supported (unsigned long type) {return type ==KVM_X86_DEFAULT_VM || (type ==KVM_X86_PROTECTED_VM &&IS_ENABLED (CONFIG_KVM_PROTECTED_VM ) && tdp_enabled); } +EXPORT_SYMBOL_GPL (__kvm_is_vm_type_supported); + +static bool kvm_is_vm_type_supported (unsigned long type) +{ +return static_call (kvm_x86_is_vm_type_supported)(type); +}int kvm_vm_ioctl_check_extension (struct kvm *kvm,long ext) {
@@ -4618,6 +4624,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r =BIT (KVM_X86_DEFAULT_VM );if (kvm_is_vm_type_supported (KVM_X86_PROTECTED_VM )) r |=BIT (KVM_X86_PROTECTED_VM ); +if (kvm_is_vm_type_supported (KVM_X86_TDX_VM )) + r |=BIT (KVM_X86_TDX_VM );break ;default :break ;
arch/x86/kvm/x86.h +2 -0
@@ -9,6 +9,8 @@#include "kvm_cache_regs.h" #include "kvm_emulate.h" +bool __kvm_is_vm_type_supported (unsigned long type); +struct kvm_caps {/* control of guest tsc rate supported? */ bool has_tsc_control;
Signature · 1 line
2.25.1
RFC PATCH v3 5/11 KVM: x86/mmu: Pass around full 64-bit error code for the KVM page fault
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Because the full 64-bit error code, or more info about the fault, for the KVM page fault will be needed for protected VM, TDX and SEV-SNP, update kvm_mmu_do_page_fault() to accept the 64-bit value so it can pass it to the callbacks.
The upper 32 bits of error code are discarded at kvm_mmu_page_fault() by lower_32_bits(). Now it's passed down as full 64 bits. Currently two hardware defined bits, PFERR_GUEST_FINAL_MASK and PFERR_GUEST_PAGE_MASK, and one software defined bit, PFERR_IMPLICIT_ACCESS, is defined.
PFERR_IMPLICIT_ACCESS:
commit 4f4aa80e3b88 ("KVM: X86: Handle implicit supervisor access with SMAP") introduced a software defined bit PFERR_IMPLICIT_ACCESS at bit 48 to indicate implicit access for SMAP with instruction emulator. Concretely emulator_read_std() and emulator_write_std() set the bit. permission_fault() checks the bit as smap implicit access. The vendor page fault handler shouldn't pass the bit to kvm_mmu_page_fault().
PFERR_GUEST_FINAL_MASK and PFERR_GUEST_PAGE_MASK:
commit 147277540bbc ("kvm: svm: Add support for additional SVM NPF error codes") introduced them to optimize the nested page fault handling. Other code path doesn't use the bits. Those two bits can be safely passed down without functionality change.
The accesses of fault->error_code are as follows
- FNAME(page_fault): PFERR_IMPLICIT_ACCESS shouldn't be passed down. PFERR_GUEST_FINAL_MASK and PFERR_GUEST_PAGE_MASK aren't used.
- kvm_mmu_page_fault(): explicit mask with PFERR_RSVD_MASK, and PFERR_NESTED_GUEST_PAGE is used outside of the masking upper 32 bits.
- mmutrace: change u32 -> u64
- pgprintk(): change %x -> %llx
No functional change is intended. This is a preparation to pass on more info with page fault error code.
---
Changes v2 -> v3:
- Make depends on a patch to clear PFERR_IMPLICIT_ACCESS
- drop clearing the upper 32 bit, instead just pass whole 64 bits
- update commit message to mention about PFERR_IMPLICIT_ACCESS and PFERR_NESTED_GUEST_PAGE
Changes v1 -> v2:
- no change
--- arch/x86/kvm/mmu/mmu.c | 5 ++--- arch/x86/kvm/mmu/mmu_internal.h | 4 ++-- arch/x86/kvm/mmu/mmutrace.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- 4 files changed, 6 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -3
@@ -4510,7 +4510,7 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultstatic int nonpaging_page_fault (struct kvm_vcpu *vcpu,struct kvm_page_fault *fault) { -pgprintk ("%s: gva %lx error %x\n" , __func__, fault->addr, fault->error_code); +pgprintk ("%s: gva %llx error %llx\n" , __func__, fault->addr, fault->error_code);/* This path builds a PAE pagetable, we can map 2mb pages at maximum. */ fault->max_level =PG_LEVEL_2M ;
@@ -5820,8 +5820,7 @@ int noinline kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 err }if (r ==RET_PF_INVALID ) { - r =kvm_mmu_do_page_fault (vcpu, cr2_or_gpa, -lower_32_bits (error_code), false, + r =kvm_mmu_do_page_fault (vcpu, cr2_or_gpa, error_code, false, &emulation_type);if (KVM_BUG_ON (r ==RET_PF_INVALID , vcpu->kvm))return -EIO ;
arch/x86/kvm/mmu/mmu_internal.h +2 -2
@@ -191,7 +191,7 @@ static inline bool is_nx_huge_page_enabled(struct kvm *kvm)struct kvm_page_fault {/* arguments to kvm_mmu_do_page_fault. */ const gpa_t addr; -const u32 error_code; +const u64 error_code;const bool prefetch;/* Derived from error_code. */
@@ -283,7 +283,7 @@ enum { };static inline int kvm_mmu_do_page_fault (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa, -u32 err,bool prefetch,int *emulation_type) +u64 err,bool prefetch,int *emulation_type) {struct kvm_page_fault fault = { .addr = cr2_or_gpa,
arch/x86/kvm/mmu/mmutrace.h +1 -1
@@ -261,7 +261,7 @@ TRACE_EVENT(TP_STRUCT__entry (__field (int ,vcpu_id )__field (gpa_t, cr2_or_gpa) -__field (u32, error_code) +__field (u64, error_code) __field(u64 *, sptep)__field (u64, old_spte) __field(u64, new_spte)
arch/x86/kvm/mmu/paging_tmpl.h +1 -1
@@ -758,7 +758,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultstruct guest_walker walker;int r; -pgprintk ("%s: addr %lx err %x\n" , __func__, fault->addr, fault->error_code); +pgprintk ("%s: addr %llx err %llx\n" , __func__, fault->addr, fault->error_code);WARN_ON_ONCE (fault->is_tdp);/*
Signature · 1 line
2.25.1
RFC PATCH v3 6/11 KVM: x86: Introduce PFERR_GUEST_ENC_MASK to indicate fault is private
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add two PFERR codes to designate that the page fault is private and that it requires looking up memory attributes. The vendor kvm page fault handler should set PFERR_GUEST_ENC_MASK bit based on their fault information. It may or may not use the hardware value directly or parse the hardware value to set the bit.
For KVM_X86_PROTECTED_VM, ask memory attributes for the fault privateness.
---
Changes v2 -> v3:
- Revive PFERR_GUEST_ENC_MASK
- rename struct kvm_page_fault::is_private => private
- Add check KVM_X86_PROTECTED_VM
Changes v1 -> v2:
- Introduced fault type and replaced is_private with fault_type.
- Add kvm_get_fault_type() to encapsulate the difference.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu/mmu.c | 14 +++++++++----- arch/x86/kvm/mmu/mmu_internal.h | 16 ++++++++++++++-- 3 files changed, 25 insertions(+), 7 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -255,6 +255,7 @@ enum x86_intercept_stage;#define PFERR_SGX_BIT 15#define PFERR_GUEST_FINAL_BIT 32#define PFERR_GUEST_PAGE_BIT 33 +#define PFERR_GUEST_ENC_BIT 34#define PFERR_IMPLICIT_ACCESS_BIT 48#define PFERR_PRESENT_MASK BIT(PFERR_PRESENT_BIT)
@@ -266,6 +267,7 @@ enum x86_intercept_stage;#define PFERR_SGX_MASK BIT(PFERR_SGX_BIT)#define PFERR_GUEST_FINAL_MASK BIT_ULL(PFERR_GUEST_FINAL_BIT)#define PFERR_GUEST_PAGE_MASK BIT_ULL(PFERR_GUEST_PAGE_BIT) +#define PFERR_GUEST_ENC_MASK BIT_ULL(PFERR_GUEST_ENC_BIT)#define PFERR_IMPLICIT_ACCESS BIT_ULL(PFERR_IMPLICIT_ACCESS_BIT)#define PFERR_NESTED_GUEST_PAGE (PFERR_GUEST_PAGE_MASK | \
arch/x86/kvm/mmu/mmu.c +9 -5
@@ -3228,7 +3228,7 @@ void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault*/ fault->req_level =__kvm_mmu_max_mapping_level (vcpu->kvm, slot, fault->gfn, fault->max_level, - fault->is_private); + fault->private);if (fault->req_level ==PG_LEVEL_4K || fault->huge_page_disallowed)return ;
@@ -4328,7 +4328,7 @@ static int kvm_do_memory_fault_exit(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { vcpu->run->exit_reason =KVM_EXIT_MEMORY_FAULT ; -if (fault->is_private) +if (fault->private) vcpu->run->memory.flags =KVM_MEMORY_EXIT_FLAG_PRIVATE ;else vcpu->run->memory.flags =0 ;
@@ -4386,10 +4386,14 @@ static int __kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *faultreturn RET_PF_EMULATE ; } -if (fault->is_private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) -return kvm_do_memory_fault_exit (vcpu, fault); +if (fault->private !=kvm_mem_is_private (vcpu->kvm, fault->gfn)) { +if (vcpu->kvm->arch.vm_type ==KVM_X86_PROTECTED_VM ) +return RET_PF_RETRY ; +else +return kvm_do_memory_fault_exit (vcpu, fault); + } -if (fault->is_private) +if (fault->private)return kvm_faultin_pfn_private (vcpu, fault); async = false;
arch/x86/kvm/mmu/mmu_internal.h +14 -2
@@ -200,10 +200,10 @@ struct kvm_page_fault {const bool present;const bool rsvd;const bool user; +const bool private;/* Derived from mmu and global state. */ const bool is_tdp; -const bool is_private;const bool nx_huge_page_workaround_enabled;/*
@@ -282,6 +282,18 @@ enum { RET_PF_SPURIOUS, }; +static inline bool kvm_is_fault_private (struct kvm *kvm,gpa_t gpa,u64 error_code) +{ +/* +* This is racy with mmu_seq. If we hit a race, it would result in a +* spurious KVM_EXIT_MEMORY_FAULT. +*/ +if (kvm->arch.vm_type ==KVM_X86_PROTECTED_VM ) +return kvm_mem_is_private (kvm,gpa_to_gfn (gpa)); + +return error_code &PFERR_GUEST_ENC_MASK ; +} +static inline int kvm_mmu_do_page_fault (struct kvm_vcpu *vcpu,gpa_t cr2_or_gpa,u64 err,bool prefetch,int *emulation_type) {
@@ -293,6 +305,7 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, .present = err & PFERR_PRESENT_MASK, .rsvd = err & PFERR_RSVD_MASK, .user = err & PFERR_USER_MASK, + .private = kvm_is_fault_private(vcpu->kvm, cr2_or_gpa, err), .prefetch = prefetch, .is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault), .nx_huge_page_workaround_enabled =
@@ -301,7 +314,6 @@ static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, .max_level = KVM_MAX_HUGEPAGE_LEVEL, .req_level = PG_LEVEL_4K, .goal_level = PG_LEVEL_4K, - .is_private = kvm_mem_is_private(vcpu->kvm, cr2_or_gpa >> PAGE_SHIFT), };int r;
Signature · 1 line
2.25.1
RFC PATCH v3 7/11 KVM: x86: Export the kvm_zap_gfn_range() for the SNP use
In reply to [email protected]From: Brijesh Singh <[email protected]>
While resolving the RMP page fault, there may be cases where the page level between the RMP entry and TDP does not match and the 2M RMP entry must be split into 4K RMP entries. Or a 2M TDP page need to be broken into multiple of 4K pages.
To keep the RMP and TDP page level in sync, zap the gfn range after splitting the pages in the RMP entry. The zap should force the TDP to gets rebuilt with the new page level.
---
Changes v2 -> v3:
- Newly added
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/mmu.h | 2 -- arch/x86/kvm/mmu/mmu.c | 1 + 3 files changed, 3 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1842,6 +1842,8 @@ void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm,void kvm_mmu_zap_all (struct kvm *kvm);void kvm_mmu_invalidate_mmio_sptes (struct kvm *kvm,u64 gen);void kvm_mmu_change_mmu_pages (struct kvm *kvm,unsigned long kvm_nr_mmu_pages); +void kvm_zap_gfn_range (struct kvm *kvm,gfn_t gfn_start,gfn_t gfn_end); +int load_pdptrs (struct kvm_vcpu *vcpu,unsigned long cr3);
arch/x86/kvm/mmu.h +0 -2
@@ -235,8 +235,6 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,return -(u32 )fault & errcode; } -void kvm_zap_gfn_range (struct kvm *kvm,gfn_t gfn_start,gfn_t gfn_end); -int kvm_arch_write_log_dirty (struct kvm_vcpu *vcpu);int kvm_mmu_post_init_vm (struct kvm *kvm);
arch/x86/kvm/mmu/mmu.c +1 -0
@@ -6727,6 +6727,7 @@ static bool kvm_mmu_zap_collapsible_spte(struct kvm *kvm,return need_tlb_flush; } +EXPORT_SYMBOL_GPL (kvm_zap_gfn_range);static void kvm_rmap_zap_collapsible_sptes (struct kvm *kvm,const struct kvm_memory_slot *slot)
Signature · 1 line
2.25.1
RFC PATCH v3 8/11 KVM: Fix set_mem_attr ioctl when error case
In reply to [email protected]From: Isaku Yamahata <[email protected]>
kvm_vm_ioctl_set_mem_attributes() discarded an error code of xa_err() unconditionally. If an error occurred at the beginning, return error.
---
Changes v2 -> v3:
- Newly added
--- virt/kvm/kvm_main.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
virt/kvm/kvm_main.c +8 -4
@@ -2423,6 +2423,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,gfn_t start, end;unsigned long i;void *entry; +int err =0 ;/* flags is currently not used. */ if (attrs->flags)
@@ -2447,14 +2448,17 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,KVM_MMU_UNLOCK (kvm);for (i = start; i < end; i++) { -if (xa_err (xa_store (&kvm->mem_attr_array, i, entry, -GFP_KERNEL_ACCOUNT ))) + err =xa_err (xa_store (&kvm->mem_attr_array, i, entry, +GFP_KERNEL_ACCOUNT )); +if (err)break ; }KVM_MMU_LOCK (kvm); -if (i > start) +if (i > start) { + err =0 ;kvm_mem_attrs_changed (kvm, attrs->attributes, start, i); + }kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm);
@@ -2463,7 +2467,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; - return 0; + return err; }#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
Signature · 1 line
2.25.1
Sean Christopherson
maintainerRe: [RFC PATCH v3 08/11] KVM: Fix set_mem_attr ioctl when error case
Quoted 49 lines · On Wed, Jun 28, 2023, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
kvm_vm_ioctl_set_mem_attributes()discarded an error code ofxa_err()unconditionally. If an error occurred at the beginning, return error.Fixes 3779c214835b ("KVM: Introduce per-page memory attributes")Signed-off-by Isaku Yamahata <[email protected]>---
Changes v2 -> v3:
- Newly added--- virt/kvm/kvm_main.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)virt/kvm/kvm_main.c +7
-3@@ -2423,6 +2423,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,gfn_t start, end;unsigned long i;void *entry; +int err =0 ;/* flags is currently not used. */ if (attrs->flags)@@ -2447,14 +2448,17 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,KVM_MMU_UNLOCK (kvm);for (i = start; i < end; i++) { -if (xa_err (xa_store (&kvm->mem_attr_array, i, entry, -GFP_KERNEL_ACCOUNT ))) + err =xa_err (xa_store (&kvm->mem_attr_array, i, entry, +GFP_KERNEL_ACCOUNT )); +if (err)break ; }KVM_MMU_LOCK (kvm); -if (i > start) +if (i > start) { + err =0 ;kvm_mem_attrs_changed (kvm, attrs->attributes, start, i); + }kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm);
virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2470
Quoted 5 lines
virt/kvm/kvm_main.c +1
-1@@ -2463,7 +2467,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; -return 0 ; +return err;
Aha! Idea (stolen from commit afb2acb2e3a3 ("KVM: Fix vcpu_array[0] races")). Rather than deal with a potential error partway through the updates, reserve all xarray entries head of time. That way the ioctl() is all-or-nothing, e.g. KVM doesn't need to update the address+size to capture progress, and userspace doesn't have to retry (which is probably pointless anyways since failure to allocate an xarray entry likely means the system/cgroup is under intense memory pressure).
Assuming it works (compile tested only), I'll squash this:
include/uapi/linux/kvm.h +1 -1
@@ -2278,7 +2278,7 @@ struct kvm_s390_zpci_op {/* Available with KVM_CAP_MEMORY_ATTRIBUTES */ #define KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES _IOR(KVMIO, 0xd2, __u64) -#define KVM_SET_MEMORY_ATTRIBUTES _IOWR(KVMIO, 0xd3, struct kvm_memory_attributes) +#define KVM_SET_MEMORY_ATTRIBUTES _IOW(KVMIO, 0xd3, struct kvm_memory_attributes)struct kvm_memory_attributes {__u64 address;
virt/kvm/kvm_main.c +19 -9
@@ -2425,6 +2425,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,gfn_t start, end;unsigned long i;void *entry; +int r;/* flags is currently not used. */ if (attrs->flags)
@@ -2439,18 +2440,32 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, start = attrs->address >>PAGE_SHIFT ; end = (attrs->address + attrs->size -1 +PAGE_SIZE ) >>PAGE_SHIFT ; +if (WARN_ON_ONCE (start == end)) +return -EINVAL ; + entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ;mutex_lock (&kvm->slots_lock); +/* +* Reserve memory ahead of time to avoid having to deal with failures +* partway through setting the new attributes. +*/ +for (i = start; i < end; i++) { + r =xa_reserve (&kvm->mem_attr_array, i,GFP_KERNEL_ACCOUNT ); +if (r) + goto out_unlock; + } +KVM_MMU_LOCK (kvm);kvm_mmu_invalidate_begin (kvm);kvm_mmu_invalidate_range_add (kvm, start, end);KVM_MMU_UNLOCK (kvm);for (i = start; i < end; i++) { -if (xa_err (xa_store (&kvm->mem_attr_array, i, entry, -GFP_KERNEL_ACCOUNT ))) + r =xa_err (xa_store (&kvm->mem_attr_array, i, entry, +GFP_KERNEL_ACCOUNT )); +if (KVM_BUG_ON (r, kvm))break ; }
@@ -2460,12 +2475,10 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm); +out_unlock:mutex_unlock (&kvm->slots_lock); - attrs->address = i <<PAGE_SHIFT ; - attrs->size = (end - i) <<PAGE_SHIFT ; - - return 0; + return r; }#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
@@ -5078,9 +5091,6 @@ static long kvm_vm_ioctl(struct file *filp, goto out; r =kvm_vm_ioctl_set_mem_attributes (kvm, &attrs); - -if (!r &©_to_user (argp, &attrs,sizeof (attrs))) - r = -EFAULT ; break; }#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
Re: [RFC PATCH v3 08/11] KVM: Fix set_mem_attr ioctl when error case
Quoted 115 lines · On Thu, 13 Jul 2023 15:03:54 -0700 Sean Christopherson <[email protected]>…
Quoted 49 lines · On Wed, Jun 28, 2023, [email protected] wrote:
From: Isaku Yamahata <[email protected]>
kvm_vm_ioctl_set_mem_attributes()discarded an error code ofxa_err()unconditionally. If an error occurred at the beginning, return error.Fixes 3779c214835b ("KVM: Introduce per-page memory attributes")Signed-off-by Isaku Yamahata <[email protected]>---
Changes v2 -> v3:
- Newly added--- virt/kvm/kvm_main.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)virt/kvm/kvm_main.c +7
-3@@ -2423,6 +2423,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,gfn_t start, end;unsigned long i;void *entry; +int err =0 ;/* flags is currently not used. */ if (attrs->flags)@@ -2447,14 +2448,17 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,KVM_MMU_UNLOCK (kvm);for (i = start; i < end; i++) { -if (xa_err (xa_store (&kvm->mem_attr_array, i, entry, -GFP_KERNEL_ACCOUNT ))) + err =xa_err (xa_store (&kvm->mem_attr_array, i, entry, +GFP_KERNEL_ACCOUNT )); +if (err)break ; }KVM_MMU_LOCK (kvm); -if (i > start) +if (i > start) { + err =0 ;kvm_mem_attrs_changed (kvm, attrs->attributes, start, i); + }kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm);virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2470
Quoted 5 lines
virt/kvm/kvm_main.c +1
-1@@ -2463,7 +2467,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, attrs->address = i <<PAGE_SHIFT ; attrs->size = (end - i) <<PAGE_SHIFT ; -return 0 ; +return err;Aha! Idea (stolen from commit
afb2acb2e3a3("KVM: Fix vcpu_array[0] races")). Rather than deal with a potential error partway through the updates, reserve all xarray entries head of time. That way theioctl()is all-or-nothing, e.g. KVM doesn't need to update the address+size to capture progress, and userspace doesn't have to retry (which is probably pointless anyways since failure to allocate an xarray entry likely means the system/cgroup is under intense memory pressure).Assuming it works (compile tested only), I'll squash this:
include/uapi/linux/kvm.h +1
-1@@ -2278,7 +2278,7 @@ struct kvm_s390_zpci_op {/* Available with KVM_CAP_MEMORY_ATTRIBUTES */ #define KVM_GET_SUPPORTED_MEMORY_ATTRIBUTES _IOR(KVMIO, 0xd2, __u64) -#define KVM_SET_MEMORY_ATTRIBUTES _IOWR(KVMIO, 0xd3, struct kvm_memory_attributes) +#define KVM_SET_MEMORY_ATTRIBUTES _IOW(KVMIO, 0xd3, struct kvm_memory_attributes)struct kvm_memory_attributes {__u64 address;virt/kvm/kvm_main.c +14
-0@@ -2425,6 +2425,7 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,gfn_t start, end;unsigned long i;void *entry; +int r;/* flags is currently not used. */ if (attrs->flags)@@ -2439,18 +2440,32 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, start = attrs->address >>PAGE_SHIFT ; end = (attrs->address + attrs->size -1 +PAGE_SIZE ) >>PAGE_SHIFT ; +if (WARN_ON_ONCE (start == end)) +return -EINVAL ; + entry = attrs->attributes ?xa_mk_value (attrs->attributes) :NULL ;mutex_lock (&kvm->slots_lock); +/* +* Reserve memory ahead of time to avoid having to deal with failures +* partway through setting the new attributes. +*/ +for (i = start; i < end; i++) { + r =xa_reserve (&kvm->mem_attr_array, i,GFP_KERNEL_ACCOUNT ); +if (r) + goto out_unlock; + } +KVM_MMU_LOCK (kvm);kvm_mmu_invalidate_begin (kvm);kvm_mmu_invalidate_range_add (kvm, start, end);KVM_MMU_UNLOCK (kvm);
virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2471
Quoted 10 lines
virt/kvm/kvm_main.c +3
-2for (i = start; i < end; i++) { -if (xa_err (xa_store (&kvm->mem_attr_array, i, entry, -GFP_KERNEL_ACCOUNT ))) + r =xa_err (xa_store (&kvm->mem_attr_array, i, entry, +GFP_KERNEL_ACCOUNT )); +if (KVM_BUG_ON (r, kvm))break ; }
IIUC, If an error happenes here, we should bail out and call xa_release()? Or the code below (which is not shown here) still changes the memory attrs partially.
Quoted 26 lines
virt/kvm/kvm_main.c +2
-7@@ -2460,12 +2475,10 @@ static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm,kvm_mmu_invalidate_end (kvm);KVM_MMU_UNLOCK (kvm); +out_unlock:mutex_unlock (&kvm->slots_lock); - attrs->address = i <<PAGE_SHIFT ; - attrs->size = (end - i) <<PAGE_SHIFT ; - - return 0; + return r; }#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */ @@ -5078,9 +5091,6 @@ static long kvm_vm_ioctl(struct file *filp, goto out; r =kvm_vm_ioctl_set_mem_attributes (kvm, &attrs); - -if (!r &©_to_user (argp, &attrs,sizeof (attrs))) - r = -EFAULT ; break; }#endif /* CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES */
Sean Christopherson
maintainerRe: [RFC PATCH v3 08/11] KVM: Fix set_mem_attr ioctl when error case
Quoted 27 lines · Zhi Wang
Quoted 14 lines · On Thu, 13 Jul 2023 15:03:54 -0700 Sean Christopherson <[email protected]>…
virt/kvm/kvm_main.c +10
-0+/* +* Reserve memory ahead of time to avoid having to deal with failures +* partway through setting the new attributes. +*/ +for (i = start; i < end; i++) { + r =xa_reserve (&kvm->mem_attr_array, i,GFP_KERNEL_ACCOUNT ); +if (r) + goto out_unlock; + } +KVM_MMU_LOCK (kvm);kvm_mmu_invalidate_begin (kvm);kvm_mmu_invalidate_range_add (kvm, start, end);virt/kvm/kvm_main.c static int kvm_vm_ioctl_set_mem_attributes(struct kvm *kvm, · line 2470
Quoted 10 lines
virt/kvm/kvm_main.c +3
-2KVM_MMU_UNLOCK (kvm);for (i = start; i < end; i++) { -if (xa_err (xa_store (&kvm->mem_attr_array, i, entry, -GFP_KERNEL_ACCOUNT ))) + r =xa_err (xa_store (&kvm->mem_attr_array, i, entry, +GFP_KERNEL_ACCOUNT )); +if (KVM_BUG_ON (r, kvm))break ; }
Quoted 3 lines
IIUC, If an error happenes here, we should bail out and call
xa_release()? Or the code below (which is not shown here) still changes the memory attrs partially.
I'm pretty sure we want to continue on. The VM is dead (killed by KVM_BUG_ON()), so the attributes as seen by userspace and/or the VM don't matter. What does matter is that KVM's internal state is consistent, e.g. that KVM doesn't have shared SPTEs while the attributes say a GFN is private. That might not matter for teardown, but I can't think of any reason not to tidy up.
And there can also be other ioctls() in flight. KVM_REQ_VM_DEAD ensures vCPU can't enter the guest, and vm->vm_dead ensures no new ioctls() cant start, but neither of those guarantees there aren't other tasks doing KVM things.
Regardless, we definitely don't need to do xa_release(). The VM is dead and all its memory will be reclaimed soon enough. And there's no guarantee xa_release() will actually be able to free anything, e.g. already processed entries won't be freed, nor will any entries that existed _before_ the ioctl() was invoked. Not to mention that the xarray probably isn't consuming much memory, relatively speaking. I.e. in the majority of scenarios, it's likely preferable to get out and destroy the VM asap.
RFC PATCH v3 9/11 KVM: Add new members to struct kvm_gfn_range to operate on
In reply to [email protected]From: Isaku Yamahata <[email protected]>
Add new members to strut kvm_gfn_range to indicate which mapping (private-vs-shared) to operate on. only_private and only_shared. Update mmu notifier, set memory attributes ioctl or KVM gmem callback to initialize them.
It was premature for set_memory_attributes ioctl to call kvm_unmap_gfn_range(). Instead, let kvm_arch_ste_memory_attributes() handle it and add a new x86 vendor callback to react to memory attribute change. [1]
- If it's from the mmu notifier, zap shared pages only
- If it's from the KVM gmem, zap private pages only
- If setting memory attributes, vendor callback checks new attributes and make decisions.
SNP would do nothing and handle it later with gmem callback TDX callback would do as follows.
When it converts pages to shared, zap private pages only. When it converts pages to private, zap shared pages only.
TDX needs to know which mapping to operate on. Shared-EPT vs. Secure-EPT. The following sequence to convert the GPA to private doesn't work for TDX because the page can already be private.
1) Update memory attributes to private in memory attributes xarray
2) Zap the GPA range irrespective of private-or-shared. Even if the page is already private, zap the entry.
3) EPT violation on the GPA
4) Populate the GPA as private
The page is zeroed, and the guest has to accept the page again.
In step 2, TDX wants to zap only shared pages and skip private ones.
[1] https://lore.kernel.org/all/[email protected]/
---
Changes v2 -> v3:
- Drop the KVM_GFN_RANGE flags
- Updated struct kvm_gfn_range
- Change kvm_arch_set_memory_attributes() to return bool for flush
- Added set_memory_attributes x86 op for vendor backends
- Refined commit message to describe TDX care concretely
Changes v1 -> v2:
- consolidate KVM_GFN_RANGE_FLAGS_GMEM_{PUNCH_HOLE, RELEASE} into KVM_GFN_RANGE_FLAGS_GMEM.
- Update the commit message to describe TDX more. Drop SEV_SNP.
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 20 +++++++++++++++----- include/linux/kvm_host.h | 13 ++++++++----- virt/kvm/guest_mem.c | 2 ++ virt/kvm/kvm_main.c | 12 ++++++------ 6 files changed, 35 insertions(+), 16 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -125,6 +125,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) KVM_X86_OP_OPTIONAL(vm_move_enc_context_from) KVM_X86_OP_OPTIONAL(guest_memory_reclaimed) +KVM_X86_OP_OPTIONAL_RET0(set_memory_attributes) KVM_X86_OP(get_msr_feature) KVM_X86_OP(can_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -1534,6 +1534,8 @@ static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical)return dest_mode_logical ?APIC_DEST_LOGICAL :APIC_DEST_PHYSICAL ; } +struct kvm_gfn_range ; +struct kvm_x86_ops {const char *name;
@@ -1716,6 +1718,7 @@ struct kvm_x86_ops {int (*vm_copy_enc_context_from)(struct kvm *kvm,unsigned int source_fd);int (*vm_move_enc_context_from)(struct kvm *kvm,unsigned int source_fd);void (*guest_memory_reclaimed)(struct kvm *kvm); +bool (*set_memory_attributes)(struct kvm *kvm,struct kvm_gfn_range *range);int (*get_msr_feature)(struct kvm_msr_entry *entry);
arch/x86/kvm/mmu/mmu.c +15 -5
@@ -7327,11 +7327,13 @@ static bool hugepage_has_attrs(struct kvm *kvm, struct kvm_memory_slot *slot,return true; } -void kvm_arch_set_memory_attributes (struct kvm *kvm, -struct kvm_memory_slot *slot, -unsigned long attrs, -gfn_t start,gfn_t end) +bool kvm_arch_set_memory_attributes (struct kvm *kvm,struct kvm_gfn_range *range) { +struct kvm_memory_slot *slot = range->slot; +unsigned long attrs = range->attributes; +gfn_t start = range->start; +gfn_t end = range->end; +bool flush = false;int level;lockdep_assert_held_write (&kvm->mmu_lock);
@@ -7342,7 +7344,13 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,* the slot if the slot will never consume the PRIVATE attribute. */ if (!kvm_slot_can_be_private (slot)) -return ; +return flush; + +if (kvm->arch.vm_type ==KVM_X86_PROTECTED_VM && + !kvm_x86_ops.set_memory_attributes) + flush =kvm_unmap_gfn_range (kvm, range); +else + flush =static_call (kvm_x86_set_memory_attributes)(kvm, range);/* * The sequence matters here: upper levels consume the result of lower
@@ -7388,6 +7396,8 @@ void kvm_arch_set_memory_attributes(struct kvm *kvm,hugepage_set_mixed (slot, gfn, level); } } + +return flush; } void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
include/linux/kvm_host.h +8 -5
@@ -260,7 +260,13 @@ struct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;gfn_t end; -pte_t pte; +union { +unsigned long attributes; +pte_t pte; +unsigned long callback_arg;/* needs a better name */ + }; +bool only_private; +bool only_shared;bool may_block; };bool kvm_unmap_gfn_range (struct kvm *kvm,struct kvm_gfn_range *range);
@@ -2323,10 +2329,7 @@ static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfnreturn xa_to_value (xa_load (&kvm->mem_attr_array, gfn)); } -void kvm_arch_set_memory_attributes (struct kvm *kvm, -struct kvm_memory_slot *slot, -unsigned long attrs, -gfn_t start,gfn_t end); +bool kvm_arch_set_memory_attributes (struct kvm *kvm,struct kvm_gfn_range *range);static inline bool kvm_mem_is_private (struct kvm *kvm,gfn_t gfn) {
virt/kvm/guest_mem.c +2 -0
@@ -117,6 +117,8 @@ static void kvm_gmem_invalidate_begin(struct kvm *kvm, struct kvm_gmem *gmem, .end = slot->base_gfn + index_end - slot->gmem.index, .slot = slot, .pte = __pte(0), + .only_private = true, + .only_shared = false, .may_block = true, };
virt/kvm/kvm_main.c +6 -6
@@ -604,6 +604,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm,* the second or later invocation of the handler). */ gfn_range.pte = range->pte; + gfn_range.only_private = false; + gfn_range.only_shared = true; gfn_range.may_block = range->may_block;/*
@@ -2391,7 +2393,9 @@ static void kvm_mem_attrs_changed(struct kvm *kvm, unsigned long attrs,bool flush = false;int i; - gfn_range.pte =__pte (0 ); + gfn_range.attributes = attrs; + gfn_range.only_private = false; + gfn_range.only_shared = false; gfn_range.may_block = true;for (i =0 ; i <kvm_arch_nr_memslot_as_ids (kvm); i++) {
@@ -2405,11 +2409,7 @@ static void kvm_mem_attrs_changed(struct kvm *kvm, unsigned long attrs,continue ; gfn_range.slot = slot; - flush |=kvm_unmap_gfn_range (kvm, &gfn_range); - -kvm_arch_set_memory_attributes (kvm, slot, attrs, - gfn_range.start, - gfn_range.end); + flush |=kvm_arch_set_memory_attributes (kvm, &gfn_range); } }
Signature · 1 line
2.25.1
Sean Christopherson
maintainerRe: [RFC PATCH v3 09/11] KVM: Add new members to struct kvm_gfn_range to operate on
+Yu
Quoted 6 lines · On Wed, Jun 28, 2023, [email protected] wrote:
void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
include/linux/kvm_host.h +0
-0
include/linux/kvm_host.h struct kvm_gfn_range { · line 267
Quoted 9 lines
include/linux/kvm_host.h +5
-1@@ -260,7 +260,13 @@ struct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;gfn_t end; -pte_t pte; +union { +unsigned long attributes; +pte_t pte; +unsigned long callback_arg;/* needs a better name */ + };
Making the union needs to be done in a separate patch. And coming back to this with fresh eyes, I think it makes sense to give the union a name. I think an anonymous union is actually worse in the long run, and there aren't _that_ many instances to update. E.g. that way a single build-time assertion can capture all uses, and it makes it more obvious that the usage is poking into a union.
I'll post a patch separately so that it can be picked up for the MGLRU series (and maybe even merged ahead of both).
Re: [RFC PATCH v3 09/11] KVM: Add new members to struct kvm_gfn_range to operate on
Quoted 26 lines · Sean Christopherson
+Yu
Quoted 6 lines · On Wed, Jun 28, 2023, [email protected] wrote:
void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
include/linux/kvm_host.h +0
-0include/linux/kvm_host.h struct kvm_gfn_range { · line 267
Quoted 9 lines
include/linux/kvm_host.h +5
-1@@ -260,7 +260,13 @@ struct kvm_gfn_range {struct kvm_memory_slot *slot;gfn_t start;gfn_t end; -pte_t pte; +union { +unsigned long attributes; +pte_t pte; +unsigned long callback_arg;/* needs a better name */ + };Making the union needs to be done in a separate patch. And coming back to this with fresh eyes, I think it makes sense to give the union a name. I think an anonymous union is actually worse in the long run, and there aren't _that_ many instances to update. E.g. that way a single build-time assertion can capture all uses, and it makes it more obvious that the usage is poking into a union.
Quoted 2 lines
I'll post a patch separately so that it can be picked up for the MGLRU series (and maybe even merged ahead of both).
Thanks a lot, Sean. And sorry for having not addressed your comments on v2 -- I'm wrapping up a few other projects and will be focusing on addressing all pending comments in a couple of weeks.
RFC PATCH v3 10/11 KVM: x86: Add gmem hook for initializing private memory
In reply to [email protected]From: Michael Roth <[email protected]>
All gmem pages are expected to be 'private' as defined by a particular arch/platform. Platforms like SEV-SNP require additional operations to move these pages into a private state, so implement a hook that can be used to prepare this memory prior to mapping it into a guest.
In the case of SEV-SNP, whether or not a 2MB page can be mapped via a 2MB mapping in the guest's nested page table depends on whether or not any subpages within the range have already been initialized as private in the RMP table, so this hook will also be used by the KVM MMU to clamp the maximum mapping size accordingly.
---
Changes v2 -> v3:
- Newly added
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 3 +++ arch/x86/kvm/mmu/mmu.c | 11 ++++++++++- 3 files changed, 14 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -135,6 +135,7 @@ KVM_X86_OP(msr_filter_changed) KVM_X86_OP(complete_emulated_msr) KVM_X86_OP(vcpu_deliver_sipi_vector) KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons); +KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) #undef KVM_X86_OP #undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +3 -0
@@ -1738,6 +1738,9 @@ struct kvm_x86_ops {* Returns vCPU specific APICv inhibit reasons */ unsigned long (*vcpu_get_apicv_inhibit_reasons)(struct kvm_vcpu *vcpu); + +int (*gmem_prepare)(struct kvm *kvm,struct kvm_memory_slot *slot, +kvm_pfn_t pfn,gfn_t gfn,u8 *max_level); };struct kvm_x86_nested_ops {
arch/x86/kvm/mmu/mmu.c +10 -1
@@ -4341,6 +4341,7 @@ static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) {int order, r; +u8 max_level;if (!kvm_slot_can_be_private (fault->slot))return kvm_do_memory_fault_exit (vcpu, fault);
@@ -4349,7 +4350,15 @@ static int kvm_faultin_pfn_private(struct kvm_vcpu *vcpu,if (r)return r; - fault->max_level =min (kvm_max_level_for_order (order), fault->max_level); + max_level =kvm_max_level_for_order (order); + r =static_call (kvm_x86_gmem_prepare)(vcpu->kvm, fault->slot, fault->pfn, + fault->gfn, &max_level); +if (r) { +kvm_release_pfn_clean (fault->pfn); +return r; + } + + fault->max_level =min (max_level, fault->max_level); fault->map_writable = !(fault->slot->flags &KVM_MEM_READONLY );return RET_PF_CONTINUE ; }
Signature · 1 line
2.25.1
RFC PATCH v3 11/11 KVM: x86: Add gmem hook for invalidating private memory
In reply to [email protected]From: Michael Roth <[email protected]>
TODO: add a CONFIG option that can be to completely skip arch invalidation loop and avoid __weak references for arch/platforms that don't need an additional invalidation hook.
In some cases, like with SEV-SNP, guest memory needs to be updated in a platform-specific manner before it can be safely freed back to the host. Add hooks to wire up handling of this sort when freeing memory in response to FALLOC_FL_PUNCH_HOLE operations.
Also issue invalidations of all allocated pages when releasing the gmem file so that the pages are not left in an unusable state when they get freed back to the host.
---
Changes v2 -> v3:
- Newly added
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 6 ++++ include/linux/kvm_host.h | 3 ++ virt/kvm/guest_mem.c | 48 ++++++++++++++++++++++++++++-- 5 files changed, 57 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -136,6 +136,7 @@ KVM_X86_OP(complete_emulated_msr)KVM_X86_OP (vcpu_deliver_sipi_vector)KVM_X86_OP_OPTIONAL_RET0 (vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) +KVM_X86_OP_OPTIONAL(gmem_invalidate)#undef KVM_X86_OP#undef KVM_X86_OP_OPTIONAL
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1741,6 +1741,7 @@ struct kvm_x86_ops {int (*gmem_prepare)(struct kvm *kvm,struct kvm_memory_slot *slot,kvm_pfn_t pfn,gfn_t gfn,u8 *max_level); +void (*gmem_invalidate)(struct kvm *kvm,kvm_pfn_t start,kvm_pfn_t end); };struct kvm_x86_nested_ops {
arch/x86/kvm/x86.c +6 -0
@@ -13260,6 +13260,12 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu) }EXPORT_SYMBOL_GPL (kvm_arch_no_poll); +#ifdef CONFIG_KVM_PRIVATE_MEM +void kvm_arch_gmem_invalidate (struct kvm *kvm,kvm_pfn_t start,kvm_pfn_t end) +{ +static_call_cond (kvm_x86_gmem_invalidate)(kvm, start, end); +} +#endif int kvm_spec_ctrl_test_value (u64 value) {
include/linux/kvm_host.h +3 -0
@@ -2346,6 +2346,7 @@ static inline bool kvm_mem_is_private(struct kvm *kvm, gfn_t gfn)#ifdef CONFIG_KVM_PRIVATE_MEM int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,kvm_pfn_t *pfn,int *order); +void kvm_arch_gmem_invalidate (struct kvm *kvm,kvm_pfn_t start,kvm_pfn_t end);#else static inline int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn,
@@ -2354,6 +2355,8 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,KVM_BUG_ON (1 , kvm);return -EIO ; } + +void kvm_arch_gmem_invalidate (struct kvm *kvm,kvm_pfn_t start,kvm_pfn_t end) { }#endif /* CONFIG_KVM_PRIVATE_MEM */ #endif
virt/kvm/guest_mem.c +46 -2
74 lines not on this page · raw message
Signature · 1 line
2.25.1
Sean Christopherson
maintainerQuoted 4 lines · On Wed, Jun 28, 2023, [email protected] wrote:
Isaku Yamahata (8):
KVM: selftests: Fixtest_add_overlapping_private_memory_regions()
KVM: selftests: Fixguest_memfd()
KVM: selftests: x86: typo inprivate_mem_conversions_test.c
Folded these fixes into the guest_memfd RFC[*].
Quoted 1 line
KVM: Fix
set_mem_attrioctl when error case
And this one too.
Quoted 1 line
KVM: Add new members to struct
kvm_gfn_rangeto operate on
And also included patches that achieve this (and will also post them separately as non-RFC, mainly to coordinate with MGLRU).