Enable Sub-Page Write Protection Support
Subsystems (5)
v12 cover Enable Sub-Page Write Protection Support
EPT-Based Sub-Page write Protection(SPP) allows Virtual Machine Monitor(VMM) specify write-permission for guest physical memory at a sub-page(128 byte) granularity. When SPP works, HW enforces write-access check for sub-pages within a protected 4KB page.
The feature targets to provide fine-grained memory protection for usages such as memory guard and VM introspection etc.
SPP is active when the "sub-page write protection" (bit 23) is set in Secondary VM-Execution Controls. The feature is backed with a Sub-Page Permission Table(SPPT). The subpage permission vector is stored in the leaf entry of SPPT. The root page is referenced via a Sub-Page Permission Table Pointer (SPPTP) in VMCS.
To enable SPP for guest memory, the guest page should be first mapped in a 4KB EPT entry, then set SPP bit 61 of the corresponding entry. While HW walks EPT, it traverses SPPT with the gpa to look up the sub-page permission vector within SPPT leaf entry. If the corresponding bit is set, write to sub-page is permitted, otherwise, SPP induced EPT violation is generated.
This patch serial passed SPP function test and selftest on Ice-Lake platform.
Please refer to the SPP introduction document in this patch set and Intel SDM for details:
Intel SDM:
https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf
Patch 1: Documentation for SPP and related API.
Patch 2: Put MMU/SSP shared definitions to a new mmu_internal.h file. Patch 3: SPPT setup functions
Patch 4: Functions to {get|set}_subpage permission
Patch 5: Introduce user-space SPP IOCTLs
Patch 6: Handle SPP induced vmexit and EPT violation
Patch 7: Enable Lazy mode SPP protection
Patch 8: Re-enable SPP if EPT mapping changes
Patch 9: Enable SPP in instruction emulation
Patch 10: Initialize SPP related data structures.
Patch 11: selftest for SPP.
Change logs:
v12:
1. Put MMU/SPP shared definitions/prototypes into a new mmu_internal.h per maintainers' comments.
2. Changed fast_page_fault()'s return type from bool to RET_PF_* per Paolo's suggestion.
3. Re-allocate SPPT root page if the root is allocated in an early spp_init() but is freed accompanied with EPT root page release.
The issue is reported by Stefan Sicleru <[email protected]>.
4. Other refactor per above changes.
5. Rebased patches to 5.7-rc5.
6. Fixed a virtual address mapping issue of selftest.
v11:
1. Refactored patches Per Sean's review feedback.
2. Added HW/KVM capabilities check before initializes SPP.
3. Combined a few functions having similar usages.
4. Removed unecessary functions in kvm_x86_ops.
5. Other code fix according to testing.
v10:
1. Cleared SPP active flag on VM resetting.
2. Added trancepoints on subpage setup and SPP induced vmexits.
3. Fixed a few code issues reported by Intel test robot.
v9:
1. Added SPP protection check in pte prefetch case.
2. Flushed EPT rmap to remove existing mappings of the target gfns.
3. Modified documentation to reflect recent changes.
4. Other minor code refactor.
v8:
1. Changed ioctl interface definition per Paolo's comments.
2. Replaced SPP_INIT ioctl funciton with KVM_ENABLE_CAP.
3. Removed SPP bit from X86 feature word.
4. Returned instruction length to user-space when SPP induced EPT violation happens, this is to provide flexibility to use SPP, revert write or track write.
5. Modified selftest application and added into this serial.
6. Simplified SPP permission vector check.
7. Moved spp.c and spp.h to kvm/mmu folder.
8. Other code fix according to Paolo's feedback and testing.
v7:
1. Configured all available protected pages once SPP induced vmexit happens since there's no PRESENT bit in SPPT leaf entry.
2. Changed SPP protection check flow in tdp_page_fault().
3. Code refactor and minior fixes.
v6:
1. Added SPP protection patch for emulation cases per Jim's review.
2. Modified documentation and added API description per Jim's review.
3. Other minior changes suggested by Jim.
v5:
1. Enable SPP support for Hugepage(1GB/2MB) to extend application.
2. Make SPP miss vm-exit handler as the unified place to set up SPPT.
3. If SPP protected pages are access-tracked or dirty-page-tracked, store SPP flag in reserved address bit, restore it in fast_page_fault() handler.
4. Move SPP specific functions to vmx/spp.c and vmx/spp.h
5. Rebased code to kernel v5.3
6. Other change suggested by KVM community.
v4:
1. Modified documentation to make it consistent with patches.
2. Allocated SPPT root page in init_spp() instead of vmx_set_cr3() to avoid SPPT miss error.
3. Added back co-developers and sign-offs.
v3:
1. Rebased patches to kernel 5.1 release
2. Deferred SPPT setup to EPT fault handler if the page is not available while set_subpage() is being called.
3. Added init IOCTL to reduce extra cost if SPP is not used.
4. Refactored patch structure, cleaned up cross referenced functions.
5. Added code to deal with memory swapping/migration/shrinker cases.
v1:
1. Rebased to 4.20-rc1
2. Move VMCS change to a separated patch.
3. Code refine and Bug fix
Yang Weijiang (11):
Documentation: Add EPT based Subpage Protection and related APIs
mmu: spp: Add a new header file to put definitions shared by MMU and
SPP
mmu: spp: Implement SPPT setup functions
mmu: spp: Implement functions to {get|set}_subpage permission
x86: spp: Introduce user-space SPP IOCTLs
vmx: spp: Handle SPP induced vmexit and EPT violation
mmu: spp: Enable Lazy mode SPP protection
mmu: spp: Re-enable SPP protection when EPT mapping changes
x86: spp: Add SPP protection check in instruction emulation
vmx: spp: Initialize SPP bitmap and SPP protection
kvm: selftests: selftest for Sub-Page protection
Documentation/virt/kvm/api.rst | 38 ++ Documentation/virtual/kvm/spp_kvm.txt | 179 +++++ arch/x86/include/asm/kvm_host.h | 11 +- arch/x86/include/asm/vmx.h | 10 + arch/x86/include/asm/vmxfeatures.h | 1 + arch/x86/include/uapi/asm/vmx.h | 2 + arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/mmu.h | 9 +- arch/x86/kvm/mmu/mmu.c | 287 ++++---- arch/x86/kvm/mmu/spp.c | 621 ++++++++++++++++++ arch/x86/kvm/mmu/spp.h | 39 ++ arch/x86/kvm/mmu_internal.h | 147 +++++ arch/x86/kvm/mmutrace.h | 10 +- arch/x86/kvm/trace.h | 66 ++ arch/x86/kvm/vmx/capabilities.h | 5 + arch/x86/kvm/vmx/vmx.c | 110 ++++ arch/x86/kvm/x86.c | 135 ++++ include/uapi/linux/kvm.h | 17 + tools/testing/selftests/kvm/Makefile | 1 + tools/testing/selftests/kvm/lib/kvm_util.c | 1 + tools/testing/selftests/kvm/x86_64/spp_test.c | 235 +++++++ 21 files changed, 1793 insertions(+), 133 deletions(-) create mode 100644 Documentation/virtual/kvm/spp_kvm.txt create mode 100644 arch/x86/kvm/mmu/spp.c create mode 100644 arch/x86/kvm/mmu/spp.h create mode 100644 arch/x86/kvm/mmu_internal.h create mode 100644 tools/testing/selftests/kvm/x86_64/spp_test.c
Signature · 1 line
2.17.2
PATCH v12 1/11 Documentation: Add EPT based Subpage Protection and related APIs
--- Documentation/virt/kvm/api.rst | 38 ++++++ Documentation/virtual/kvm/spp_kvm.txt | 179 ++++++++++++++++++++++++++ 2 files changed, 217 insertions(+) create mode 100644 Documentation/virtual/kvm/spp_kvm.txt
Documentation/virt/kvm/api.rst +38 -0
@@ -4690,6 +4690,44 @@ KVM_PV_VM_VERIFY Verify the integrity of the unpacked image. Only if this succeeds, KVM is allowed to start protected VCPUs. +4.126 KVM_SUBPAGES_GET_ACCESS + +Architectures: x86 +Type: vm ioctl +Parameters: struct kvm_subpage_info (in/out) +Returns: 0 on success, < 0 on error + +#define KVM_SUBPAGE_MAX_PAGES 512 +struct kvm_subpage { + __u64 gfn_base; /* the first page gfn of the contiguous pages */ + __u32 npages; /* number of 4K pages */ + __u32 flags; /* reserved to 0 now */ + __u32 access_map[0]; /* start place of bitmap array */ +}; + +This ioctl fetches subpage permission from contiguous pages starting with +gfn. npages is the number of contiguous pages to fetch. access_map contains permission +vectors fetched for all the pages. + +4.127 KVM_SUBPAGES_SET_ACCESS + +Architectures: x86 +Type: vm ioctl +Parameters: struct kvm_subpage_info (in/out) +Returns: 0 on success, < 0 on error + +#define KVM_SUBPAGE_MAX_PAGES 512 +struct kvm_subpage { + __u64 gfn_base; /* the first page gfn of the contiguous pages */ + __u32 npages; /* number of 4K pages */ + __u32 flags; /* reserved to 0 now */ + __u32 access_map[0]; /* start place of bitmap array */ +}; + +This ioctl sets subpage permission for contiguous pages starting with gfn. npages is +the number of contiguous pages to set. access_map contains permission vectors for all the +pages. Since during execution of the ioctl, it holds mmu_lock, so limits the MAX pages +to 512 to reduce the impact to EPT. 5. The kvm_run structure ========================
Documentation/virtual/kvm/spp_kvm.txt new file +179 -0
179 lines not on this page · raw message
Signature · 1 line
2.17.2
PATCH v12 2/11 mmu: spp: Add a new header file to put definitions shared by MMU and SPP
In reply to Yang WeijiangSPP re-uses many MMU data structures, macros and functions etc. To separate spp.c as a new compilation unit, put these shared parts into a new mmu_internal.h file.
--- arch/x86/kvm/mmu.h | 7 -- arch/x86/kvm/mmu/mmu.c | 140 ++++++++++------------------------ arch/x86/kvm/mmu_internal.h | 145 ++++++++++++++++++++++++++++++++++++ 3 files changed, 185 insertions(+), 107 deletions(-) create mode 100644 arch/x86/kvm/mmu_internal.h
arch/x86/kvm/mmu.h +0 -7
@@ -22,8 +22,6 @@#define PT_ACCESSED_MASK (1ULL << PT_ACCESSED_SHIFT)#define PT_DIRTY_SHIFT 6#define PT_DIRTY_MASK (1ULL << PT_DIRTY_SHIFT) -#define PT_PAGE_SIZE_SHIFT 7 -#define PT_PAGE_SIZE_MASK (1ULL << PT_PAGE_SIZE_SHIFT)#define PT_PAT_MASK (1ULL << 7)#define PT_GLOBAL_MASK (1ULL << 8)#define PT64_NX_SHIFT 63
@@ -38,11 +36,6 @@#define PT32_DIR_PSE36_MASK \ (((1ULL << PT32_DIR_PSE36_SIZE) - 1) << PT32_DIR_PSE36_SHIFT) -#define PT64_ROOT_5LEVEL 5 -#define PT64_ROOT_4LEVEL 4 -#define PT32_ROOT_LEVEL 2 -#define PT32E_ROOT_LEVEL 3 -static inline u64 rsvd_bits (int s,int e) {if (e < s)
arch/x86/kvm/mmu/mmu.c +40 -100
409 lines not on this page · raw message
arch/x86/kvm/mmu_internal.h new file +145 -0
145 lines not on this page · raw message
Signature · 1 line
2.17.2
PATCH v12 3/11 mmu: spp: Implement SPPT setup functions
In reply to Yang WeijiangSPPT(Sub-Page Permission Table) is a 4-level structure similar to EPT. If SPP is enabled in secondary execution control, WP and SPP bit(61) are set in a 4KB EPT leaf entry, then SPPT is traversed with the gfn, the leaf entry of SPPT contains the permission vector, every subpage within the 4KB page owns one bit.
SPPT setup is similar to that of EPT therefore a lot of EPT helpers are re-used in spp.c. To make least change to mmu.c and keep the patch clean meanwhile, spp.c is embedded at the end of mmu.c.
Specific to SPPT:
1)The leaf entry contains a 64-bit permission vector. Subpage is
128B each so 4KB/128B = 32bits are required for write permission. The even bits(2*i) corrrespond to write-permission to subpage(i), if it's 1, subpage(i) is writable, otherwise, it's write-protected. The odd bits are reserved and must be 0.
2)When permission vectors are updated, it first flushes the corresponding
entry at SPPT L2E by making the entry invalid so that following SPPT walk can trigger SPP-miss handling, there the permission vectors are rebuilt.
spp.c is built as a separate object file now per previous patch.
--- arch/x86/include/asm/kvm_host.h | 5 +- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/mmu/spp.c | 129 ++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/spp.h | 5 ++ 4 files changed, 139 insertions(+), 2 deletions(-) create mode 100644 arch/x86/kvm/mmu/spp.c create mode 100644 arch/x86/kvm/mmu/spp.h
arch/x86/include/asm/kvm_host.h +4 -1
@@ -274,7 +274,8 @@ union kvm_mmu_page_role { unsigned smap_andnot_wp:1; unsigned ad_disabled:1; unsigned guest_mode:1; - unsigned :6; + unsigned spp:1; + unsigned reserved:5;/* * This is left at the top of the word so that
@@ -982,6 +983,8 @@ struct kvm_arch {struct kvm_pmu_event_filter *pmu_event_filter;struct task_struct *nx_lpage_recovery_thread; + +hpa_t sppt_root; };struct kvm_vm_stat {
arch/x86/kvm/Makefile +1 -1
@@ -15,7 +15,7 @@ kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \ i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \ - hyperv.o debugfs.o mmu/mmu.o mmu/page_track.o + hyperv.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spp.o kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o
arch/x86/kvm/mmu/spp.c new file +129 -0
129 lines not on this page · raw message
arch/x86/kvm/mmu/spp.h new file +5 -0
@@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __KVM_X86_VMX_SPP_H +#define __KVM_X86_VMX_SPP_H + +#endif /* __KVM_X86_VMX_SPP_H */
Signature · 1 line
2.17.2
PATCH v12 4/11 mmu: spp: Implement functions to {get|set}_subpage permission
In reply to Yang WeijiangUserspace application can {set|get} subpage permission via IOCTLs if SPP has been initialized.
Steps for set_permission:
1)Store the permission vectors to SPP bitmap buffer. 2)Flush existing hugepage mapping in rmap so as to avoid
stale mapping.
3)Walk EPT to check if gfn->pfn 4KB mapping is there,
mark the existing entry as WP and SPP protected.
4)Zap the entry if gfn->pfn is hugepage mapping so that
following memory access can trigger EPT page_fault() to set up SPP protection.
--- arch/x86/include/asm/kvm_host.h | 3 + arch/x86/kvm/mmu.h | 2 + arch/x86/kvm/mmu/spp.c | 244 ++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/spp.h | 12 ++ arch/x86/kvm/mmu_internal.h | 2 + arch/x86/kvm/trace.h | 22 +++ arch/x86/kvm/x86.c | 1 + include/uapi/linux/kvm.h | 8 ++ 8 files changed, 294 insertions(+)
arch/x86/include/asm/kvm_host.h +3 -0
@@ -723,6 +723,7 @@ struct kvm_vcpu_arch {unsigned nmi_pending;/* NMI queued after currently running handler */ bool nmi_injected;/* Trying to inject an NMI this entry */ bool smi_pending;/* SMI queued after currently running handler */ +bool spp_pending;/* SPP has been requested, need to update VMCS */ struct kvm_mtrr mtrr_state;u64 pat;
@@ -829,6 +830,7 @@ struct kvm_lpage_info {struct kvm_arch_memory_slot {struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES ]; +u32 *subpage_wp_info;struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES -1 ];unsigned short *gfn_track[KVM_PAGE_TRACK_MAX ]; };
@@ -985,6 +987,7 @@ struct kvm_arch {struct task_struct *nx_lpage_recovery_thread;hpa_t sppt_root; +bool spp_active; };struct kvm_vm_stat {
arch/x86/kvm/mmu.h +2 -0
@@ -26,6 +26,8 @@#define PT_GLOBAL_MASK (1ULL << 8)#define PT64_NX_SHIFT 63#define PT64_NX_MASK (1ULL << PT64_NX_SHIFT) +#define PT_SPP_SHIFT 61 +#define PT_SPP_MASK (1ULL << PT_SPP_SHIFT)#define PT_PAT_SHIFT 7#define PT_DIR_PAT_SHIFT 12
arch/x86/kvm/mmu/spp.c +244 -0
265 lines not on this page · raw message
arch/x86/kvm/mmu/spp.h +12 -0
@@ -2,4 +2,16 @@#ifndef __KVM_X86_VMX_SPP_H#define __KVM_X86_VMX_SPP_H +#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1) + +int kvm_spp_get_permission (struct kvm *kvm,u64 gfn,u32 npages, +u32 *access_map); +int kvm_spp_set_permission (struct kvm *kvm,u64 gfn,u32 npages, +u32 *access_map); +int kvm_spp_mark_protection (struct kvm *kvm,u64 gfn,u32 access); + +int kvm_spp_setup_structure (struct kvm_vcpu *vcpu, +u32 access_map,gfn_t gfn); +u32 *gfn_to_subpage_wp_info (struct kvm_memory_slot *slot,gfn_t gfn); +#endif /* __KVM_X86_VMX_SPP_H */
arch/x86/kvm/mmu_internal.h +2 -0
@@ -124,6 +124,8 @@ void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep);u64 __get_spte_lockless (u64 *sptep); +void pte_list_remove (struct kvm_rmap_head *rmap_head,u64 *sptep); +u64 mmu_spte_get_lockless (u64 *sptep);unsigned kvm_page_table_hashfn (gfn_t gfn);
arch/x86/kvm/trace.h +22 -0
@@ -1539,6 +1539,28 @@ TRACE_EVENT(kvm_nested_vmenter_failed, __print_symbolic(__entry->err, VMX_VMENTER_INSTRUCTION_ERRORS)) ); +TRACE_EVENT (kvm_spp_set_subpages, + TP_PROTO(struct kvm_vcpu *vcpu, gfn_t gfn, u32 access), +TP_ARGS (vcpu, gfn, access), + +TP_STRUCT__entry ( + __field(int, vcpu_id) + __field(gfn_t, gfn) +__field (u32, access) + ), + +TP_fast_assign ( + __entry->vcpu_id = vcpu->vcpu_id; + __entry->gfn = gfn; + __entry->access = access; + ), + +TP_printk ("vcpu %d gfn %llx access %x" , + __entry->vcpu_id, + __entry->gfn, + __entry->access) +); +#endif /* _TRACE_KVM_H */ #undef TRACE_INCLUDE_PATH
arch/x86/kvm/x86.c +1 -0
@@ -10601,3 +10601,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_unaccelerated_access);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_incomplete_ipi);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_ga_log);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_apicv_update_request); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_spp_set_subpages);
include/uapi/linux/kvm.h +8 -0
@@ -102,6 +102,14 @@ struct kvm_userspace_memory_region {__u64 userspace_addr;/* start of the userspace allocated memory */ }; +/* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */ +struct kvm_subpage { +__u64 gfn_base;/* the first page gfn of the contiguous pages */ +__u32 npages;/* number of 4K pages */ +__u32 flags;/* reserved to 0 now */ +__u32 access_map[0 ];/* start place of bitmap array */ +}; +/* * The bit 0 ~ bit 15 of kvm_memory_region::flags are visible for userspace, * other bits are reserved for kvm internal use which are defined in
Signature · 1 line
2.17.2
PATCH v12 5/11 x86: spp: Introduce user-space SPP IOCTLs
In reply to Yang WeijiangUserspace application, e.g., QEMU or VMI, must initialize SPP before {gets|sets} SPP subpages.
--- arch/x86/kvm/mmu/spp.c | 79 ++++++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/spp.h | 11 ++++- arch/x86/kvm/x86.c | 88 ++++++++++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 2 + 4 files changed, 179 insertions(+), 1 deletion(-)
arch/x86/kvm/mmu/spp.c +79 -0
@@ -205,6 +205,46 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,return ret; } +int spp_flush_sppt (struct kvm *kvm,u64 gfn_base,u32 npages) +{ +struct kvm_shadow_walk_iterator iter; +struct kvm_vcpu *vcpu; +gfn_t gfn = gfn_base; +gfn_t gfn_end = gfn_base + npages -1 ; +u64 spde; +int count; +bool flush = false; + + vcpu =kvm_get_vcpu (kvm,0 ); +if (!VALID_PAGE (vcpu->kvm->arch.sppt_root)) +return -EFAULT ; + +for (; gfn <= gfn_end; gfn++) { +for_each_shadow_spp_entry (vcpu, (u64 )gfn <<PAGE_SHIFT , iter) { +if (!is_shadow_present_pte (*iter.sptep)) +break ; + +if (iter.level !=PT_DIRECTORY_LEVEL ) +continue ; + + spde = *iter.sptep; + spde &= ~PT_PRESENT_MASK ; +spp_spte_set (iter.sptep, spde); + count =kvm_spp_level_pages (gfn, gfn_end, +PT_DIRECTORY_LEVEL ); + flush = true; +if (count >= npages) + goto out; + gfn += count; +break ; + } + } +out: +if (flush) +kvm_flush_remote_tlbs (kvm); +return 0 ; +} +int kvm_spp_get_permission (struct kvm *kvm,u64 gfn,u32 npages,u32 *access_map) {
@@ -371,3 +411,42 @@ int kvm_spp_mark_protection(struct kvm *kvm, u64 gfn, u32 access) }return ret; } + +int kvm_vm_ioctl_get_subpages (struct kvm *kvm, +u64 gfn, +u32 npages, +u32 *access_map) +{ +int ret; + +mutex_lock (&kvm->slots_lock); + ret =kvm_spp_get_permission (kvm, gfn, npages, access_map); +mutex_unlock (&kvm->slots_lock); + +return ret; +} + +int kvm_vm_ioctl_set_subpages (struct kvm *kvm, +u64 gfn, +u32 npages, +u32 *access_map) +{ +int ret; + +spin_lock (&kvm->mmu_lock); + ret =spp_flush_sppt (kvm, gfn, npages); +spin_unlock (&kvm->mmu_lock); + +if (ret <0 ) +return ret; + +mutex_lock (&kvm->slots_lock); +spin_lock (&kvm->mmu_lock); + + ret =kvm_spp_set_permission (kvm, gfn, npages, access_map); + +spin_unlock (&kvm->mmu_lock); +mutex_unlock (&kvm->slots_lock); + +return ret; +}
arch/x86/kvm/mmu/spp.h +10 -1
@@ -3,15 +3,24 @@#define __KVM_X86_VMX_SPP_H#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1) +#define KVM_SUBPAGE_MAX_PAGES 512int kvm_spp_get_permission (struct kvm *kvm,u64 gfn,u32 npages,u32 *access_map);int kvm_spp_set_permission (struct kvm *kvm,u64 gfn,u32 npages,u32 *access_map);int kvm_spp_mark_protection (struct kvm *kvm,u64 gfn,u32 access); - +int kvm_vm_ioctl_get_subpages (struct kvm *kvm, +u64 gfn, +u32 npages, +u32 *access_map); +int kvm_vm_ioctl_set_subpages (struct kvm *kvm, +u64 gfn, +u32 npages, +u32 *access_map);int kvm_spp_setup_structure (struct kvm_vcpu *vcpu,u32 access_map,gfn_t gfn);u32 *gfn_to_subpage_wp_info (struct kvm_memory_slot *slot,gfn_t gfn); +int spp_flush_sppt (struct kvm *kvm,u64 gfn_base,u32 npages);#endif /* __KVM_X86_VMX_SPP_H */
arch/x86/kvm/x86.c +88 -0
100 lines not on this page · raw message
include/uapi/linux/kvm.h +2 -0
@@ -1280,6 +1280,8 @@ struct kvm_vfio_spapr_tce { struct kvm_userspace_memory_region)#define KVM_SET_TSS_ADDR _IO(KVMIO, 0x47)#define KVM_SET_IDENTITY_MAP_ADDR _IOW(KVMIO, 0x48, __u64) +#define KVM_SUBPAGES_GET_ACCESS _IOR(KVMIO, 0x49, __u64) +#define KVM_SUBPAGES_SET_ACCESS _IOW(KVMIO, 0x4a, __u64)/* enable ucontrol for s390 */ struct kvm_s390_ucas_mapping {
Signature · 1 line
2.17.2
PATCH v12 6/11 vmx: spp: Handle SPP induced vmexit and EPT violation
In reply to Yang WeijiangIf write to subpage is prohibited, EPT violation is generated and handled in fast_page_fault().
In current implementation, SPPT setup is handled in handle_spp() handler, it's triggered when SPP is enabled in EPT leaf entry while SPPT entry is invalid.
There could be two kinds of SPP usages, one is for write-protection, the other is for access-tracking, the differece is the former keeps memory unchange while the latter just records the memory access and may let the write take effect. To fit these two cases, when SPP induced vmexit to userspace, the fault instruction length is returned, the application may take action according to the specific use-case, re-do write operation or discard it.
To make SPP operatable with dirty-logging, introduce a free bit in EPT entry to store SPP bit, after dirty-logging happened, it restores SPP bit and make entry SPP protected again so that a retry write will trigger a normal SPP induced vmexit.
--- arch/x86/include/asm/kvm_host.h | 2 + arch/x86/include/asm/vmx.h | 9 ++++ arch/x86/include/uapi/asm/vmx.h | 2 + arch/x86/kvm/mmu/mmu.c | 92 ++++++++++++++++++++++++++------- arch/x86/kvm/mmu/spp.c | 27 ++++++++++ arch/x86/kvm/mmu/spp.h | 5 ++ arch/x86/kvm/mmutrace.h | 10 ++-- arch/x86/kvm/trace.h | 44 ++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 74 ++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 2 + include/uapi/linux/kvm.h | 6 +++ 11 files changed, 248 insertions(+), 25 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1260,6 +1260,8 @@ struct kvm_x86_ops {bool (*apic_init_signal_blocked)(struct kvm_vcpu *vcpu);int (*enable_direct_tlbflush)(struct kvm_vcpu *vcpu); + +int (*get_insn_len)(struct kvm_vcpu *vcpu); };struct kvm_x86_init_ops {
arch/x86/include/asm/vmx.h +9 -0
@@ -217,6 +217,8 @@ enum vmcs_field {XSS_EXIT_BITMAP_HIGH =0x0000202D ,ENCLS_EXITING_BITMAP =0x0000202E ,ENCLS_EXITING_BITMAP_HIGH =0x0000202F , +SPPT_POINTER =0x00002030 , +SPPT_POINTER_HIGH =0x00002031 ,TSC_MULTIPLIER =0x00002032 ,TSC_MULTIPLIER_HIGH =0x00002033 , GUEST_PHYSICAL_ADDRESS = 0x00002400,
@@ -550,6 +552,13 @@ struct vmx_msr_entry {#define EPT_VIOLATION_EXECUTABLE (1 << EPT_VIOLATION_EXECUTABLE_BIT)#define EPT_VIOLATION_GVA_TRANSLATED (1 << EPT_VIOLATION_GVA_TRANSLATED_BIT) +/* +* Exit Qualifications for SPPT-Induced vmexits +*/ +#define SPPT_INDUCED_EXIT_TYPE_BIT 11 +#define SPPT_INDUCED_EXIT_TYPE (1 << SPPT_INDUCED_EXIT_TYPE_BIT) +#define SPPT_INTR_INFO_UNBLOCK_NMI INTR_INFO_UNBLOCK_NMI +/* * VM-instruction error numbers */
arch/x86/include/uapi/asm/vmx.h +2 -0
@@ -86,6 +86,7 @@#define EXIT_REASON_PML_FULL 62#define EXIT_REASON_XSAVES 63#define EXIT_REASON_XRSTORS 64 +#define EXIT_REASON_SPP 66#define EXIT_REASON_UMWAIT 67#define EXIT_REASON_TPAUSE 68
@@ -145,6 +146,7 @@ { EXIT_REASON_ENCLS, "ENCLS" }, \ { EXIT_REASON_RDSEED, "RDSEED" }, \ { EXIT_REASON_PML_FULL, "PML_FULL" }, \ + { EXIT_REASON_SPP, "SPP" }, \ { EXIT_REASON_XSAVES, "XSAVES" }, \ { EXIT_REASON_XRSTORS, "XRSTORS" }, \ { EXIT_REASON_UMWAIT, "UMWAIT" }, \
arch/x86/kvm/mmu/mmu.c +72 -20
212 lines not on this page · raw message
arch/x86/kvm/mmu/spp.c +27 -0
@@ -20,6 +20,25 @@ static void shadow_spp_walk_init(struct kvm_shadow_walk_iterator *iterator, iterator->level =PT64_ROOT_4LEVEL ; } +/* Save reserved bit for SPP armed PTE */ +void save_spp_bit (u64 *spte) +{ + *spte |=PT64_SPP_SAVED_BIT ; + *spte &= ~PT_SPP_MASK ; +} + +/* Restore reserved bit for SPP armed PTE */ +void restore_spp_bit (u64 *spte) +{ + *spte &= ~PT64_SPP_SAVED_BIT ; + *spte |=PT_SPP_MASK ; +} + +bool was_spp_armed (u64 spte) +{ +return !!(spte &PT64_SPP_SAVED_BIT ); +} +u32 *gfn_to_subpage_wp_info (struct kvm_memory_slot *slot,gfn_t gfn) {unsigned long idx;
@@ -33,6 +52,7 @@ u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn)return &slot->arch.subpage_wp_info[idx]; } +EXPORT_SYMBOL_GPL (gfn_to_subpage_wp_info); static bool __rmap_update_subpage_bit(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
@@ -204,6 +224,7 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,kvm_flush_remote_tlbs (vcpu->kvm);return ret; } +EXPORT_SYMBOL_GPL (kvm_spp_setup_structure);int spp_flush_sppt (struct kvm *kvm,u64 gfn_base,u32 npages) {
@@ -450,3 +471,9 @@ int kvm_vm_ioctl_set_subpages(struct kvm *kvm,return ret; } + +inline u64 construct_spptp (unsigned long root_hpa) +{ +return root_hpa &PAGE_MASK ; +} +EXPORT_SYMBOL_GPL (construct_spptp);
arch/x86/kvm/mmu/spp.h +5 -0
@@ -4,6 +4,7 @@#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1)#define KVM_SUBPAGE_MAX_PAGES 512 +#define MAX_ENTRIES_PER_MMUPAGE BIT(9)int kvm_spp_get_permission (struct kvm *kvm,u64 gfn,u32 npages,u32 *access_map);
@@ -22,5 +23,9 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu, u32 access_map, gfn_t gfn);u32 *gfn_to_subpage_wp_info (struct kvm_memory_slot *slot,gfn_t gfn);int spp_flush_sppt (struct kvm *kvm,u64 gfn_base,u32 npages); +void save_spp_bit (u64 *spte); +void restore_spp_bit (u64 *spte); +bool was_spp_armed (u64 spte); +u64 construct_spptp (unsigned long root_hpa);#endif /* __KVM_X86_VMX_SPP_H */
arch/x86/kvm/mmutrace.h +5 -5
@@ -245,13 +245,13 @@ TRACE_EVENT( );#define __spte_satisfied (__spte) \ - (__entry->retry && is_writable_pte(__entry->__spte)) + (__entry->ret == RET_PF_RETRY && is_writable_pte(__entry->__spte)) TRACE_EVENT( fast_page_fault, TP_PROTO(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u32 error_code, - u64 *sptep, u64 old_spte, bool retry), - TP_ARGS(vcpu, cr2_or_gpa, error_code, sptep, old_spte, retry), + u64 *sptep, u64 old_spte, int ret), + TP_ARGS(vcpu, cr2_or_gpa, error_code, sptep, old_spte, ret), TP_STRUCT__entry( __field(int, vcpu_id)
@@ -260,7 +260,7 @@ TRACE_EVENT( __field(u64 *, sptep) __field(u64, old_spte) __field(u64, new_spte) - __field(bool, retry) + __field(int, ret) ), TP_fast_assign(
@@ -270,7 +270,7 @@ TRACE_EVENT( __entry->sptep = sptep; __entry->old_spte = old_spte; __entry->new_spte = *sptep; - __entry->retry = retry; + __entry->ret = ret; ), TP_printk("vcpu %d gva %llx error_code %s sptep %p old %#llx"
arch/x86/kvm/trace.h +44 -0
50 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +74 -0
104 lines not on this page · raw message
arch/x86/kvm/x86.c +2 -0
@@ -10690,3 +10690,5 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_incomplete_ipi);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_avic_ga_log);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_apicv_update_request);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_spp_set_subpages); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_spp_induced_exit); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_spp_induced_page_fault);
include/uapi/linux/kvm.h +6 -0
@@ -244,6 +244,7 @@ struct kvm_hyperv_exit {#define KVM_EXIT_IOAPIC_EOI 26#define KVM_EXIT_HYPERV 27#define KVM_EXIT_ARM_NISV 28 +#define KVM_EXIT_SPP 29/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -401,6 +402,11 @@ struct kvm_run {struct {__u8 vector; } eoi; +/* KVM_EXIT_SPP */ +struct { +__u64 addr; +__u8 insn_len; + } spp;/* KVM_EXIT_HYPERV */ struct kvm_hyperv_exit hyperv;/* KVM_EXIT_ARM_NISV */
Signature · 1 line
2.17.2
PATCH v12 7/11 mmu: spp: Enable Lazy mode SPP protection
In reply to Yang WeijiangWhen SPP protection is set but the gfn->pfn mapping isn't there, we need to check and mark SPP protection in EPT while gfn->pfn mapping is being built, but the setup for SPPT is deferred to handle_spp() handler.
From HW's capability, SPP only works for 4KB mappings, to apply SPP protection for hugepage(2MB,1GB) cases, hugepage entries need to be zapped before SPP set up. In tdp_page_fault(), there's check for SPP protection before sets 4KB, 2MB or 1GB mapping, the target is introducing least impact to hugepage setup, i.e., falls back to most possible hugepage mapping.
--- arch/x86/kvm/mmu/mmu.c | 19 +++++++++++++++++++ arch/x86/kvm/mmu/spp.c | 43 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/spp.h | 2 ++ 3 files changed, 64 insertions(+)
arch/x86/kvm/mmu/mmu.c +19 -0
@@ -3131,6 +3131,7 @@ static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu,unsigned int access = sp->role.access;int i, ret;gfn_t gfn; +u32 *wp_bitmap; gfn =kvm_mmu_page_get_gfn (sp, start - sp->spt); slot =gfn_to_memslot_dirty_bitmap (vcpu, gfn, access &ACC_WRITE_MASK );
@@ -3144,6 +3145,13 @@ static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu,for (i =0 ; i < ret; i++, gfn++, start++) {mmu_set_spte (vcpu, start, access,0 , sp->role.level, gfn,page_to_pfn (pages[i]), true, true); +if (vcpu->kvm->arch.spp_active) { + wp_bitmap =gfn_to_subpage_wp_info (slot, gfn); +if (wp_bitmap && *wp_bitmap !=FULL_SPP_ACCESS ) +kvm_spp_mark_protection (vcpu->kvm, + gfn, + *wp_bitmap); + }put_page (pages[i]); }
@@ -3336,6 +3344,15 @@ static int __direct_map(struct kvm_vcpu *vcpu, gpa_t gpa, int write, map_writable);direct_pte_prefetch (vcpu, it.sptep); ++vcpu->stat.pf_fixed; +if (level ==PT_PAGE_TABLE_LEVEL ) { +int ret; +u32 access; + + ret =kvm_spp_get_permission (vcpu->kvm, gfn,1 , &access); +if (ret ==1 && access !=FULL_SPP_ACCESS ) +kvm_spp_mark_protection (vcpu->kvm, gfn, access); + } +return ret; }
@@ -4125,6 +4142,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,if (lpage_disallowed) max_level =PT_PAGE_TABLE_LEVEL ; +check_spp_protection (vcpu, gfn, &max_level); + r =fast_page_fault (vcpu, gpa, error_code);if (r !=RET_PF_INVALID )
arch/x86/kvm/mmu/spp.c +43 -0
49 lines not on this page · raw message
arch/x86/kvm/mmu/spp.h +2 -0
@@ -11,6 +11,8 @@ int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,int kvm_spp_set_permission (struct kvm *kvm,u64 gfn,u32 npages,u32 *access_map);int kvm_spp_mark_protection (struct kvm *kvm,u64 gfn,u32 access); +void check_spp_protection (struct kvm_vcpu *vcpu,gfn_t gfn, +int *max_level); int kvm_vm_ioctl_get_subpages(struct kvm *kvm, u64 gfn, u32 npages,
Signature · 1 line
2.17.2
PATCH v12 8/11 mmu: spp: Re-enable SPP protection when EPT mapping changes
In reply to Yang WeijiangHost page swapping/migration may change the translation in EPT leaf entry, if the target page is SPP protected, re-enable SPP protection. When SPPT mmu-page is reclaimed, no need to clear rmap as no memory-mapping is in SPPT L4E.
--- arch/x86/kvm/mmu/mmu.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
arch/x86/kvm/mmu/mmu.c +17 -0
@@ -1796,6 +1796,19 @@ static int kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, new_spte &= ~PT_WRITABLE_MASK ; new_spte &= ~SPTE_HOST_WRITEABLE ; +/* +* if it's EPT leaf entry and the physical page is +* SPP protected, then re-enable SPP protection for +* the page. +*/ +if (kvm->arch.spp_active && + level ==PT_PAGE_TABLE_LEVEL ) { +u32 *access =gfn_to_subpage_wp_info (slot, gfn); + +if (access && *access !=FULL_SPP_ACCESS ) + new_spte |=PT_SPP_MASK ; + } + new_spte =mark_spte_for_access_track (new_spte);mmu_spte_clear_track_bits (sptep);
@@ -2639,6 +2652,10 @@ static bool mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, pte = *spte;if (is_shadow_present_pte (pte)) {if (is_last_spte (pte, sp->role.level)) { +/* SPPT leaf entries don't have rmaps*/ +if (sp->role.spp && sp->role.level == +PT_PAGE_TABLE_LEVEL ) +return true;drop_spte (kvm, spte);if (is_large_pte (pte)) --kvm->stat.lpages;
Signature · 1 line
2.17.2
PATCH v12 9/11 x86: spp: Add SPP protection check in instruction emulation
In reply to Yang WeijiangIn instruction/mmio emulation cases, if the target write memroy is SPP protected, "fake" an vmexit to userspace to let application handle it.
--- arch/x86/kvm/x86.c | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+)
arch/x86/kvm/x86.c +37 -0
@@ -5788,6 +5788,37 @@ static const struct read_write_emulator_ops write_emultor = { .write = true, }; +static bool is_emulator_spp_protected (struct kvm_vcpu *vcpu, +gpa_t gpa, +unsigned int bytes) +{ +gfn_t gfn, gfn_start, gfn_end; +struct kvm *kvm = vcpu->kvm; +struct kvm_memory_slot *slot; +u32 *access; + +if (!kvm->arch.spp_active) +return false; + + gfn_start =gpa_to_gfn (gpa); + gfn_end =gpa_to_gfn (gpa + bytes); +for (gfn = gfn_start; gfn <= gfn_end; gfn++) { + slot =gfn_to_memslot (kvm, gfn); +if (slot) { + access =gfn_to_subpage_wp_info (slot, gfn); +if (access && *access !=FULL_SPP_ACCESS ) { + vcpu->run->exit_reason =KVM_EXIT_SPP ; + vcpu->run->spp.addr = gfn; + vcpu->run->spp.insn_len = + kvm_x86_ops.get_insn_len (vcpu); +return true; + } + } + } + +return false; +} + static int emulator_read_write_onepage(unsigned long addr, void *val, unsigned int bytes, struct x86_exception *exception,
@@ -5817,6 +5848,9 @@ static int emulator_read_write_onepage(unsigned long addr, void *val,return X86EMUL_PROPAGATE_FAULT ; } +if (write &&is_emulator_spp_protected (vcpu, gpa, bytes)) +return X86EMUL_UNHANDLEABLE ; +if (!ret && ops->read_write_emulate (vcpu, gpa, val, bytes))return X86EMUL_CONTINUE ;
@@ -6963,6 +6997,9 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,return 1 ;if (r ==EMULATION_FAILED ) { +if (vcpu->run->exit_reason ==KVM_EXIT_SPP ) +return 0 ; +if (reexecute_instruction (vcpu, cr2_or_gpa, write_fault_to_spt, emulation_type))return 1 ;
Signature · 1 line
2.17.2
PATCH v12 10/11 vmx: spp: Initialize SPP bitmap and SPP protection
In reply to Yang WeijiangFor each memoryslot, there's a SPP bitmap buffer allocated, every 4Byte corresponds to subpages within a 4KB page. The original default value for each 4byte is all 1s, meaning the whole 4KB page is not SPP protected, this eases following SPP protection check.
To support SPP enablement on-demand, SPP initialization can be done via KVM_ENABLE_CAP with subcode KVM_CAP_X86_SPP. KVM_SUBPAGE_MAX_PAGES is set to 512 to reduce the impact to EPT page_fault() handling because when SPP protection is configured, mmu-lock is held.
All vcpus share the same SPPT, a KVM_REQ_LOAD_CR3 request is issued to each vcpu after SPP is initialized, in handling of the request, SPPTP and VMX SPP execution control bit are configured in VMCS.
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/include/asm/vmx.h | 1 + arch/x86/include/asm/vmxfeatures.h | 1 + arch/x86/kvm/mmu/mmu.c | 21 +++++++ arch/x86/kvm/mmu/spp.c | 99 ++++++++++++++++++++++++++++++ arch/x86/kvm/mmu/spp.h | 8 ++- arch/x86/kvm/vmx/capabilities.h | 5 ++ arch/x86/kvm/vmx/vmx.c | 36 +++++++++++ arch/x86/kvm/x86.c | 7 +++ include/uapi/linux/kvm.h | 1 + 10 files changed, 179 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1262,6 +1262,7 @@ struct kvm_x86_ops {int (*enable_direct_tlbflush)(struct kvm_vcpu *vcpu);int (*get_insn_len)(struct kvm_vcpu *vcpu); +u32 (*get_spp_status)(struct kvm_vcpu *vcpu); };struct kvm_x86_init_ops {
arch/x86/include/asm/vmx.h +1 -0
@@ -70,6 +70,7 @@#define SECONDARY_EXEC_PT_CONCEAL_VMX VMCS_CONTROL_BIT(PT_CONCEAL_VMX)#define SECONDARY_EXEC_XSAVES VMCS_CONTROL_BIT(XSAVES)#define SECONDARY_EXEC_MODE_BASED_EPT_EXEC VMCS_CONTROL_BIT(MODE_BASED_EPT_EXEC) +#define SECONDARY_EXEC_SPP VMCS_CONTROL_BIT(SPP)#define SECONDARY_EXEC_PT_USE_GPA VMCS_CONTROL_BIT(PT_USE_GPA)#define SECONDARY_EXEC_TSC_SCALING VMCS_CONTROL_BIT(TSC_SCALING)#define SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE VMCS_CONTROL_BIT(USR_WAIT_PAUSE)
arch/x86/include/asm/vmxfeatures.h +1 -0
@@ -79,6 +79,7 @@#define VMX_FEATURE_PT_CONCEAL_VMX ( 2*32+ 19)/* "" Suppress VMX indicators in Processor Trace */ #define VMX_FEATURE_XSAVES ( 2*32+ 20)/* "" Enable XSAVES and XRSTORS in guest */ #define VMX_FEATURE_MODE_BASED_EPT_EXEC ( 2*32+ 22)/* "ept_mode_based_exec" Enable separate EPT EXEC bits for supervisor vs. user */ +#define VMX_FEATURE_SPP ( 2*32+ 23)/* "spp" Enable Sub-page Permission feature */ #define VMX_FEATURE_PT_USE_GPA ( 2*32+ 24)/* "" Processor Trace logs GPAs */ #define VMX_FEATURE_TSC_SCALING ( 2*32+ 25)/* Scale hardware TSC when read in guest */ #define VMX_FEATURE_USR_WAIT_PAUSE ( 2*32+ 26)/* Enable TPAUSE, UMONITOR, UMWAIT in guest */
arch/x86/kvm/mmu/mmu.c +21 -0
@@ -3680,6 +3680,11 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, (mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) {mmu_free_root_page (vcpu->kvm, &mmu->root_hpa, &invalid_list); +if (VALID_PAGE (vcpu->kvm->arch.sppt_root)) { +mmu_free_root_page (vcpu->kvm, + &vcpu->kvm->arch.sppt_root, + &invalid_list); + } } else {for (i =0 ; i <4 ; ++i)if (mmu->pae_root[i] !=0 )
@@ -3747,6 +3752,19 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)/* root_cr3 is ignored for direct MMUs. */ vcpu->arch.mmu->root_cr3 =0 ; +/* +* If spp_init() is called too early, in some cases, e.g., CR0 paging +* bit changes from disable to enable, this causes mmu root page to +* be freed together with sppt root page, then here need to re-allocate +* it. +*/ +if (vcpu->kvm->arch.spp_active && + !VALID_PAGE (vcpu->kvm->arch.sppt_root)) { + sp =kvm_spp_get_page (vcpu,0 , + vcpu->arch.mmu->shadow_root_level); + vcpu->kvm->arch.sppt_root =__pa (sp->spt); + ++sp->root_count; + }return 0 ; }
@@ -5163,6 +5181,9 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu, bool reset_roots)uint i; vcpu->arch.mmu->root_hpa =INVALID_PAGE ; +if (!vcpu->kvm->arch.spp_active) + vcpu->kvm->arch.sppt_root =INVALID_PAGE ; +for (i =0 ; i <KVM_MMU_NUM_PREV_ROOTS ; i++) vcpu->arch.mmu->prev_roots[i] =KVM_MMU_ROOT_INFO_INVALID ; }
arch/x86/kvm/mmu/spp.c +99 -0
105 lines not on this page · raw message
arch/x86/kvm/mmu/spp.h +7 -1
@@ -5,7 +5,11 @@#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1)#define KVM_SUBPAGE_MAX_PAGES 512#define MAX_ENTRIES_PER_MMUPAGE BIT(9) +#define SPP_STATUS_VMX_SUPPORT 0x1 +#define SPP_STATUS_EPT_SUPPORT 0x2 +int spp_init (struct kvm *kvm); +void kvm_spp_free_memslot (struct kvm *kvm,struct kvm_memory_slot *slot);int kvm_spp_get_permission (struct kvm *kvm,u64 gfn,u32 npages,u32 *access_map); int kvm_spp_set_permission(struct kvm *kvm, u64 gfn, u32 npages,
@@ -29,5 +33,7 @@ void save_spp_bit(u64 *spte);void restore_spp_bit (u64 *spte);bool was_spp_armed (u64 spte);u64 construct_spptp (unsigned long root_hpa); - +struct kvm_mmu_page *kvm_spp_get_page (struct kvm_vcpu *vcpu, +gfn_t gfn, +unsigned int level);#endif /* __KVM_X86_VMX_SPP_H */
arch/x86/kvm/vmx/capabilities.h +5 -0
@@ -248,6 +248,11 @@ static inline bool vmx_xsaves_supported(void)SECONDARY_EXEC_XSAVES ; } +static inline bool cpu_has_vmx_ept_spp (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl &SECONDARY_EXEC_SPP ; +} +static inline bool vmx_waitpkg_supported (void ) { return vmcs_config.cpu_based_2nd_exec_ctrl &
arch/x86/kvm/vmx/vmx.c +36 -0
@@ -117,6 +117,9 @@ module_param_named(pml, enable_pml, bool, S_IRUGO);static bool __read_mostly dump_invalid_vmcs =0 ;module_param (dump_invalid_vmcs, bool,0644 ); +/* SPP is disabled by default unless it's enabled via KVM_ENABLE_CAP. */ +static bool __read_mostly enable_spp =0 ; +#define MSR_BITMAP_MODE_X2APIC 1#define MSR_BITMAP_MODE_X2APIC_APICV 2
@@ -1408,6 +1411,17 @@ static int vmx_get_insn_len(struct kvm_vcpu *vcpu)return vmcs_read32 (VM_EXIT_INSTRUCTION_LEN ); } +static u32 vmx_get_spp_status (struct kvm_vcpu *vcpu) +{ +u32 status =0 ; + +if (cpu_has_vmx_ept_spp ()) + status |=SPP_STATUS_VMX_SUPPORT ; +if (enable_ept) + status |=SPP_STATUS_EPT_SUPPORT ; +return status; +} +static void vmx_decache_cr0_guest_bits (struct kvm_vcpu *vcpu);unsigned long vmx_get_rflags (struct kvm_vcpu *vcpu)
@@ -2394,6 +2408,7 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,SECONDARY_EXEC_RDSEED_EXITING |SECONDARY_EXEC_RDRAND_EXITING |SECONDARY_EXEC_ENABLE_PML | +SECONDARY_EXEC_SPP |SECONDARY_EXEC_TSC_SCALING |SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE | SECONDARY_EXEC_PT_USE_GPA |
@@ -2998,6 +3013,7 @@ void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long cr3)bool update_guest_cr3 = true;unsigned long guest_cr3;u64 eptp; +u64 spptp; guest_cr3 = cr3;if (enable_ept) {
@@ -3026,6 +3042,20 @@ void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long cr3)if (update_guest_cr3)vmcs_writel (GUEST_CR3 , guest_cr3); + +if (kvm->arch.spp_active &&VALID_PAGE (vcpu->kvm->arch.sppt_root)) { + spptp =construct_spptp (vcpu->kvm->arch.sppt_root); +vmcs_write64 (SPPT_POINTER , spptp); + +if (vcpu->arch.spp_pending &&cpu_has_secondary_exec_ctrls ()) { +struct vcpu_vmx *vmx =to_vmx (vcpu); + +secondary_exec_controls_setbit (vmx, +SECONDARY_EXEC_SPP ); + enable_spp =1 ; + vcpu->arch.spp_pending = false; + } + } }int vmx_set_cr4 (struct kvm_vcpu *vcpu,unsigned long cr4)
@@ -4042,6 +4072,9 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)if (!enable_pml) exec_control &= ~SECONDARY_EXEC_ENABLE_PML ; +if (!enable_spp) + exec_control &= ~SECONDARY_EXEC_SPP ; +if (vmx_xsaves_supported ()) {/* Exposing XSAVES only when XSAVE is exposed */ bool xsaves_enabled =
@@ -5900,6 +5933,8 @@ void dump_vmcs(void)pr_err ("PostedIntrVec = 0x%02x\n" ,vmcs_read16 (POSTED_INTR_NV ));if ((secondary_exec_control &SECONDARY_EXEC_ENABLE_EPT ))pr_err ("EPT pointer = 0x%016llx\n" ,vmcs_read64 (EPT_POINTER )); +if ((secondary_exec_control &SECONDARY_EXEC_SPP )) +pr_err ("SPPT pointer = 0x%016llx\n" ,vmcs_read64 (SPPT_POINTER )); n =vmcs_read32 (CR3_TARGET_COUNT );for (i =0 ; i +1 < n; i +=4 ) pr_err("CR3 target%u=%016lx target%u=%016lx\n",
@@ -7912,6 +7947,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .apic_init_signal_blocked = vmx_apic_init_signal_blocked, .get_insn_len = vmx_get_insn_len, + .get_spp_status = vmx_get_spp_status, };static __init int hardware_setup (void )
arch/x86/kvm/x86.c +7 -0
@@ -3436,6 +3436,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)case KVM_CAP_HYPERV_ENLIGHTENED_VMCS : r = kvm_x86_ops.nested_enable_evmcs !=NULL ;break ; +case KVM_CAP_X86_SPP : + r =KVM_SUBPAGE_MAX_PAGES ; +break ;default :break ; }
@@ -4884,6 +4887,9 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, kvm->arch.exception_payload_enabled = cap->args[0 ]; r =0 ;break ; +case KVM_CAP_X86_SPP : + r =spp_init (kvm); +break ;default : r = -EINVAL ;break ;
@@ -10047,6 +10053,7 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot) }kvm_page_track_free_memslot (slot); +kvm_spp_free_memslot (kvm, slot); } static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
include/uapi/linux/kvm.h +1 -0
@@ -1031,6 +1031,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_S390_VCPU_RESETS 179#define KVM_CAP_S390_PROTECTED 180#define KVM_CAP_PPC_SECURE_GUEST 181 +#define KVM_CAP_X86_SPP 182#ifdef KVM_CAP_IRQ_ROUTING
Signature · 1 line
2.17.2
PATCH v12 11/11 kvm: selftests: selftest for Sub-Page protection
In reply to Yang WeijiangSub-Page Permission(SPP) is to protect finer granularity subpages (128Byte each) within a 4KB page. It's not enabled in KVM by default, the test first initializes the SPP runtime environment with KVM_ENABLE_CAP ioctl, then sets protection with KVM_SUBPAGES_SET_ACCESS for the target guest page, check permissions with KVM_SUBPAGES_GET_ACCESS to make sure they are set as expected.
Two steps in guest code to very whether SPP is working:
1) protect all 128byte subpages, write data to each subpage to see if SPP induced EPT violation happening. 2)unprotect all subpages, again write data to each subpage to see if SPP still works or not.
--- tools/testing/selftests/kvm/Makefile | 1 + tools/testing/selftests/kvm/lib/kvm_util.c | 1 + tools/testing/selftests/kvm/x86_64/spp_test.c | 235 ++++++++++++++++++ 3 files changed, 237 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86_64/spp_test.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -60,6 +60,7 @@ TEST_GEN_PROGS_x86_64 += dirty_log_test TEST_GEN_PROGS_x86_64 += kvm_create_max_vcpus TEST_GEN_PROGS_x86_64 += steal_time +TEST_GEN_PROGS_x86_64 += x86_64/spp_test TEST_GEN_PROGS_aarch64 += clear_dirty_log_test TEST_GEN_PROGS_aarch64 += demand_paging_test TEST_GEN_PROGS_aarch64 += dirty_log_test
tools/testing/selftests/kvm/lib/kvm_util.c +1 -0
@@ -1552,6 +1552,7 @@ static struct exit_reason { {KVM_EXIT_UNKNOWN, "UNKNOWN"}, {KVM_EXIT_EXCEPTION, "EXCEPTION"}, {KVM_EXIT_IO, "IO"}, + {KVM_EXIT_SPP, "SPP"}, {KVM_EXIT_HYPERCALL, "HYPERCALL"}, {KVM_EXIT_DEBUG, "DEBUG"}, {KVM_EXIT_HLT, "HLT"},
tools/testing/selftests/kvm/x86_64/spp_test.c new file +235 -0
235 lines not on this page · raw message
Signature · 1 line
2.17.2