Enable Sub-page Write Protection Support
Subsystems (4)
v4 cover Enable Sub-page Write Protection Support
EPT-Based Sub-Page write Protection(SPP)is a HW capability which allows Virtual Machine Monitor(VMM) to specify write-permission for guest physical memory at a sub-page(128 byte) granularity. When this capability is enabled, the CPU enforces write-access check for sub-pages within a 4KB page.
The feature is targeted to provide fine-grained memory protection for usages such as device virtualization, memory check-point and VM introspection etc.
SPP is active when the "sub-page write protection" (bit 23) is 1 in Secondary VM-Execution Controls. The feature is backed with a Sub-Page Permission Table(SPPT), SPPT is referenced via a 64-bit control field called Sub-Page Permission Table Pointer (SPPTP) which contains a 4K-aligned physical address.
Right now, only 4KB physical pages are supported for SPP. To enable SPP for certain physical page, we need to first make the physical page write-protected, then set bit 61 of the corresponding EPT leaf entry. While HW walks EPT, if bit 61 is set, it traverses SPPT with the guset physical address to find out the sub-page permissions at the leaf entry. If the corresponding bit is set, write to sub-page is permitted, otherwise, SPP induced EPT vilation is generated.
Please refer to the SPP introduction document in this patch set and Intel SDM for details:
Intel SDM:
https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf
Previous patch:
https://lkml.org/lkml/2019/6/6/695
Patch 1: Introduction to SPP.
Patch 2: Add SPP related flags and control bits. Patch 3: Functions for SPPT setup.
Patch 4: Add SPP access bitmaps for memslots. Patch 5: Low level implementation of SPP operations. Patch 6: Implement User space access IOCTLs. Patch 7: Handle SPP induced VMExit and EPT violation. Patch 8: Enable lazy mode SPPT setup.
Patch 9: Handle memory remapping and reclaim.
Change logs:
V3 -> v4:
1. Modified documentation to make it consistent with patches.
2. Allocated SPPT root page in init_spp() instead of vmx_set_cr3() to avoid SPPT miss error.
3. Added back co-developers and sign-offs.
V2 -> V3:
1. Rebased patches to kernel 5.1 release
2. Deferred SPPT setup to EPT fault handler if the page is not available while set_subpage() is being called.
3. Added init IOCTL to reduce extra cost if SPP is not used.
4. Refactored patch structure, cleaned up cross referenced functions.
5. Added code to deal with memory swapping/migration/shrinker cases.
V2 -> V1:
1. Rebased to 4.20-rc1
2. Move VMCS change to a separated patch.
3. Code refine and Bug fix
Yang Weijiang (9):
Documentation: Introduce EPT based Subpage Protection
KVM: VMX: Add control flags for SPP enabling
KVM: VMX: Implement functions for SPPT paging setup
KVM: VMX: Introduce SPP access bitmap and operation functions
KVM: VMX: Add init/set/get functions for SPP
KVM: VMX: Introduce SPP user-space IOCTLs
KVM: VMX: Handle SPP induced vmexit and page fault
KVM: MMU: Enable Lazy mode SPPT setup
KVM: MMU: Handle host memory remapping and reclaim
Documentation/virtual/kvm/spp_kvm.txt | 173 ++++++++++ arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/kvm_host.h | 26 +- arch/x86/include/asm/vmx.h | 10 + arch/x86/include/uapi/asm/vmx.h | 2 + arch/x86/kernel/cpu/intel.c | 4 + arch/x86/kvm/mmu.c | 480 ++++++++++++++++++++++++++ arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/vmx/capabilities.h | 5 + arch/x86/kvm/vmx/vmx.c | 129 +++++++ arch/x86/kvm/x86.c | 141 ++++++++ include/linux/kvm_host.h | 9 + include/uapi/linux/kvm.h | 17 + 13 files changed, 997 insertions(+), 1 deletion(-) create mode 100644 Documentation/virtual/kvm/spp_kvm.txt
Signature · 1 line
2.17.2
PATCH v4 1/9 Documentation: Introduce EPT based Subpage Protection
Co-developed-by [email protected]
Signed-off-by [email protected]
Co-developed-by Yang Weijiang <[email protected]>
Signed-off-by Yang Weijiang <[email protected]>
--- Documentation/virtual/kvm/spp_kvm.txt | 173 ++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100644 Documentation/virtual/kvm/spp_kvm.txt
Documentation/virtual/kvm/spp_kvm.txt new file +173 -0
@@ -0,0 +1,173 @@ +EPT-Based Sub-Page Protection (SPP) for KVM +==================================================== + +1.Overview + EPT-based Sub-Page Protection(SPP) allows VMM to specify + fine-grained(128byte per sub-page) write-protection for guest physical + memory. When it's enabled, the CPU enforces write-access permission + for the sub-pages within a 4KB page, if corresponding bit is set in + permission vector, write to sub-page region is allowed, otherwise, + it's prevented with a EPT violation. + +2.SPP Operation + Sub-Page Protection Table (SPPT) is introduced to manage sub-page + write-access permission. + + It is active when: + a) large paging is disabled on host side. + b) "sub-page write protection" VM-execution control is 1. + c) SPP is initialized with KVM_INIT_SPP ioctl successfully. + d) Sub-page permissions are set with KVM_SUBPAGES_SET_ACCESS ioctl + successfully. see below sections for details. + + __________________________________________________________________________ + + How SPP hardware works: + __________________________________________________________________________ + + Guest write access --> GPA --> Walk EPT --> EPT leaf entry -----| + |---------------------------------------------------------------| + |-> if VMexec_control.spp && ept_leaf_entry.spp_bit (bit 61) + | + |-> <false> --> EPT legacy behavior + | + | + |-> <true> --> if ept_leaf_entry.writable + | + |-> <true> --> Ignore SPP + | + |-> <false> --> GPA --> Walk SPP 4-level table--| + | + |------------<----------get-the-SPPT-point-from-VMCS-filed-----<------| + | + Walk SPP L4E table + | + |---> if-entry-misconfiguration ------------>-------|-------<---------| + | | | + else | | + | | | + | |------------------SPP VMexit<-----------------| | + | | | + | |-> exit_qualification & sppt_misconfig --> sppt misconfig | + | | | + | |-> exit_qualification & sppt_miss --> sppt miss | + |---| | + | | + walk SPPT L3E--|--> if-entry-misconfiguration------------>------------| + | | + else | + | | + | | + walk SPPT L2E --|--> if-entry-misconfiguration-------->-------| + | | + else | + | | + | | + walk SPPT L1E --|-> if-entry-misconfiguration--->----| + | + else + | + |-> if sub-page writable + |-> <true> allow, write access + |-> <false> disallow, EPT violation + ______________________________________________________________________________ + +3.IOCTL Interfaces + + KVM_INIT_SPP: + Allocate storage for sub-page permission vectors and SPPT root page. + + KVM_SUBPAGES_GET_ACCESS: + Get sub-page write permission vectors for given continuous guest pages. + + KVM_SUBPAGES_SET_ACCESS + Set sub-pages write permission vectors for given continuous guest pages. + + /* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */ + struct kvm_subpage_info { + __u64 gfn; /* the first page gfn of the continuous pages */ + __u64 npages; /* number of 4K pages */ + __u64 *access_map; /* sub-page write-access bitmap array */ + }; + + #define KVM_SUBPAGES_GET_ACCESS _IOR(KVMIO, 0x49, __u64) + #define KVM_SUBPAGES_SET_ACCESS _IOW(KVMIO, 0x4a, __u64) + #define KVM_INIT_SPP _IOW(KVMIO, 0x4b, __u64) + +4.Set Sub-Page Permission + + * To enable SPP protection, system admin sets sub-page permission via + KVM_SUBPAGES_SET_ACCESS ioctl: + + (1) If the target 4KB pages are there, it locates EPT leaf entries + via the guest physical addresses, sets the bit 61 of the corresponding + entries to enable sub-page protection, then set up SPPT paging structure. + (2) otherwise, stores the [gfn,permission] mappings in KVM data structure. When + EPT page-fault is generated due to access to target page, it settles + EPT entry configuration together with SPPT setup, this is called lazy mode + setup. + + The SPPT paging structure format is as below: + + Format of the SPPT L4E, L3E, L2E: + | Bit | Contents | + | :----- | :------------------------------------------------------------------------| + | 0 | Valid entry when set; indicates whether the entry is present | + | 11:1 | Reserved (0) | + | N-1:12 | Physical address of 4KB aligned SPPT LX-1 Table referenced by this entry | + | 51:N | Reserved (0) | + | 63:52 | Reserved (0) | + Note: N is the physical address width supported by the processor. X is the page level + + Format of the SPPT L1E: + | Bit | Contents | + | :---- | :---------------------------------------------------------------- | + | 0+2i | Write permission for i-th 128 byte sub-page region. | + | 1+2i | Reserved (0). | + Note: 0<=i<=31 + +5.SPPT-induced VM exit + + * SPPT miss and misconfiguration induced VM exit + + A SPPT missing VM exit occurs when walk the SPPT, there is no SPPT + misconfiguration but a paging-structure entry is not + present in any of L4E/L3E/L2E entries. + + A SPPT misconfiguration VM exit occurs when reserved bits or unsupported values + are set in SPPT entry. + + *NOTE* SPPT miss and SPPT misconfigurations can occur only due to an + attempt to write memory with a guest physical address. + + * SPP permission induced VM exit + SPP sub-page permission induced violation is reported as EPT violation + thesefore causes VM exit. + +6.SPPT-induced VM exit handling + + #define EXIT_REASON_SPP 66 + + static int (*const kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = { + ... + [EXIT_REASON_SPP] = handle_spp, + ... + }; + + New exit qualification for SPPT-induced vmexits. + + | Bit | Contents | + | :---- | :---------------------------------------------------------------- | + | 10:0 | Reserved (0). | + | 11 | SPPT VM exit type. Set for SPPT Miss, cleared for SPPT Misconfig. | + | 12 | NMI unblocking due to IRET | + | 63:13 | Reserved (0) | + + In addition to the exit qualification, guest linear address and guest + physical address fields will be reported. + + * SPPT miss and misconfiguration induced VM exit + Allocate a physical page for the SPPT and set the entry correctly. + + * SPP permission induced VM exit + This kind of VM exit is left to VMI tool to handle.
Signature · 1 line
2.17.2
PATCH v4 2/9 KVM: VMX: Add control flags for SPP enabling
In reply to Yang WeijiangCheck SPP capability in MSR_IA32_VMX_PROCBASED_CTLS2, its 23-bit indicates SPP support. Mark SPP bit in CPU capabilities bitmap if it's supported.
Co-developed-by He Chen <[email protected]>
Signed-off-by He Chen <[email protected]>
Co-developed-by Zhang Yi <[email protected]>
Signed-off-by Zhang Yi <[email protected]>
Co-developed-by Yang Weijiang <[email protected]>
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/vmx.h | 1 + arch/x86/kernel/cpu/intel.c | 4 ++++ arch/x86/kvm/vmx/capabilities.h | 5 +++++ arch/x86/kvm/vmx/vmx.c | 10 ++++++++++ 5 files changed, 21 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -228,6 +228,7 @@#define X86_FEATURE_FLEXPRIORITY ( 8*32+ 2)/* Intel FlexPriority */ #define X86_FEATURE_EPT ( 8*32+ 3)/* Intel Extended Page Table */ #define X86_FEATURE_VPID ( 8*32+ 4)/* Intel Virtual Processor ID */ +#define X86_FEATURE_SPP ( 8*32+ 5)/* Intel EPT-based Sub-Page Write Protection */ #define X86_FEATURE_VMMCALL ( 8*32+15)/* Prefer VMMCALL to VMCALL */ #define X86_FEATURE_XENPV ( 8*32+16)/* "" Xen paravirtual guest */
arch/x86/include/asm/vmx.h +1 -0
@@ -81,6 +81,7 @@#define SECONDARY_EXEC_XSAVES 0x00100000#define SECONDARY_EXEC_PT_USE_GPA 0x01000000#define SECONDARY_EXEC_MODE_BASED_EPT_EXEC 0x00400000 +#define SECONDARY_EXEC_ENABLE_SPP 0x00800000#define SECONDARY_EXEC_TSC_SCALING 0x02000000#define PIN_BASED_EXT_INTR_MASK 0x00000001
arch/x86/kernel/cpu/intel.c +4 -0
@@ -476,6 +476,7 @@ static void detect_vmx_virtcap(struct cpuinfo_x86 *c)#define X86_VMX_FEATURE_PROC_CTLS2_EPT 0x00000002#define X86_VMX_FEATURE_PROC_CTLS2_VPID 0x00000020#define x86_VMX_FEATURE_EPT_CAP_AD 0x00200000 +#define X86_VMX_FEATURE_PROC_CTLS2_SPP 0x00800000u32 vmx_msr_low, vmx_msr_high, msr_ctl, msr_ctl2;u32 msr_vpid_cap, msr_ept_cap;
@@ -486,6 +487,7 @@ static void detect_vmx_virtcap(struct cpuinfo_x86 *c)clear_cpu_cap (c,X86_FEATURE_EPT );clear_cpu_cap (c,X86_FEATURE_VPID );clear_cpu_cap (c,X86_FEATURE_EPT_AD ); +clear_cpu_cap (c,X86_FEATURE_SPP );rdmsr (MSR_IA32_VMX_PROCBASED_CTLS , vmx_msr_low, vmx_msr_high); msr_ctl = vmx_msr_high | vmx_msr_low;
@@ -509,6 +511,8 @@ static void detect_vmx_virtcap(struct cpuinfo_x86 *c) }if (msr_ctl2 &X86_VMX_FEATURE_PROC_CTLS2_VPID )set_cpu_cap (c,X86_FEATURE_VPID ); +if (msr_ctl2 &X86_VMX_FEATURE_PROC_CTLS2_SPP ) +set_cpu_cap (c,X86_FEATURE_SPP ); } }
arch/x86/kvm/vmx/capabilities.h +5 -0
@@ -239,6 +239,11 @@ static inline bool cpu_has_vmx_pml(void)return vmcs_config.cpu_based_2nd_exec_ctrl &SECONDARY_EXEC_ENABLE_PML ; } +static inline bool cpu_has_vmx_ept_spp (void ) +{ +return vmcs_config.cpu_based_2nd_exec_ctrl &SECONDARY_EXEC_ENABLE_SPP ; +} +static inline bool vmx_xsaves_supported (void ) { return vmcs_config.cpu_based_2nd_exec_ctrl &
arch/x86/kvm/vmx/vmx.c +10 -0
@@ -114,6 +114,8 @@ static u64 __read_mostly host_xss;bool __read_mostly enable_pml =1 ;module_param_named (pml, enable_pml, bool,S_IRUGO ); +static bool __read_mostly spp_supported =0 ; +#define MSR_BITMAP_MODE_X2APIC 1#define MSR_BITMAP_MODE_X2APIC_APICV 2
@@ -2240,6 +2242,7 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,SECONDARY_EXEC_RDSEED_EXITING |SECONDARY_EXEC_RDRAND_EXITING |SECONDARY_EXEC_ENABLE_PML | +SECONDARY_EXEC_ENABLE_SPP |SECONDARY_EXEC_TSC_SCALING |SECONDARY_EXEC_PT_USE_GPA | SECONDARY_EXEC_PT_CONCEAL_VMX |
@@ -3895,6 +3898,9 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)if (!enable_pml) exec_control &= ~SECONDARY_EXEC_ENABLE_PML ; +if (!spp_supported) + exec_control &= ~SECONDARY_EXEC_ENABLE_SPP ; +if (vmx_xsaves_supported ()) {/* Exposing XSAVES only when XSAVE is exposed */ bool xsaves_enabled =
@@ -7470,6 +7476,10 @@ static __init int hardware_setup(void)if (!cpu_has_vmx_flexpriority ()) flexpriority_enabled =0 ; +if (cpu_has_vmx_ept_spp () && enable_ept && +boot_cpu_has (X86_FEATURE_SPP )) + spp_supported =1 ; +if (!cpu_has_virtual_nmis ()) enable_vnmi =0 ;
Signature · 1 line
2.17.2
PATCH v4 3/9 KVM: VMX: Implement functions for SPPT paging setup
In reply to Yang WeijiangSPPT is a 4-level paging structure similar to EPT, when SPP is kicked for target physical page, bit 61 of the corresponding EPT enty will be flaged, then SPPT is traversed with the gfn to build up entries, the leaf entry of SPPT contains the access bitmap for subpages inside the target 4KB physical page, one bit per 128-byte subpage.
SPPT entries are set up in below cases:
1. the EPT faulted page is SPP protected.
2. SPP mis-config induced vmexit is handled.
3. User configures SPP protected pages via SPP IOCTLs.
Co-developed-by He Chen <[email protected]>
Signed-off-by He Chen <[email protected]>
Co-developed-by Zhang Yi <[email protected]>
Signed-off-by Zhang Yi <[email protected]>
Co-developed-by Yang Weijiang <[email protected]>
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 7 +- arch/x86/kvm/mmu.c | 207 ++++++++++++++++++++++++++++++++ arch/x86/kvm/mmu.h | 1 + include/linux/kvm_host.h | 3 + 4 files changed, 217 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +6 -1
@@ -271,7 +271,8 @@ union kvm_mmu_page_role { unsigned smap_andnot_wp:1; unsigned ad_disabled:1; unsigned guest_mode:1; - unsigned :6; + unsigned spp:1; + unsigned reserved:5;/* * This is left at the top of the word so that
@@ -1407,6 +1408,10 @@ int kvm_emulate_hypercall(struct kvm_vcpu *vcpu);int kvm_mmu_page_fault (struct kvm_vcpu *vcpu,gva_t gva,u64 error_code,void *insn,int insn_len); + +int kvm_mmu_setup_spp_structure (struct kvm_vcpu *vcpu, +u32 access_map,gfn_t gfn); +void kvm_mmu_invlpg (struct kvm_vcpu *vcpu,gva_t gva);void kvm_mmu_invpcid_gva (struct kvm_vcpu *vcpu,gva_t gva,unsigned long pcid);void kvm_mmu_new_cr3 (struct kvm_vcpu *vcpu,gpa_t new_cr3,bool skip_tlb_flush);
arch/x86/kvm/mmu.c +207 -0
255 lines not on this page · raw message
arch/x86/kvm/mmu.h +1 -0
@@ -26,6 +26,7 @@#define PT_PAGE_SIZE_MASK (1ULL << PT_PAGE_SIZE_SHIFT)#define PT_PAT_MASK (1ULL << 7)#define PT_GLOBAL_MASK (1ULL << 8) +#define PT_SPP_MASK (1ULL << 61)#define PT64_NX_SHIFT 63#define PT64_NX_MASK (1ULL << PT64_NX_SHIFT)
include/linux/kvm_host.h +3 -0
@@ -847,6 +847,9 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu);bool kvm_arch_vcpu_in_kernel (struct kvm_vcpu *vcpu);int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu); +struct kvm_mmu_page *kvm_mmu_get_spp_page (struct kvm_vcpu *vcpu, +gfn_t gfn,unsigned int level); +#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* * All architectures that want to use vzalloc currently also
Signature · 1 line
2.17.2
PATCH v4 4/9 KVM: VMX: Introduce SPP access bitmap and operation functions
In reply to Yang WeijiangCreate access bitmap for SPP subpages, 4KB/128B = 32bits, for each 4KB physical page, 32bits are required. The bitmap can be easily accessed with a gfn. The initial access bitmap for each physical page is 0xFFFFFFFF, meaning SPP is not enabled for the subpages.
Co-developed-by He Chen <[email protected]>
Signed-off-by He Chen <[email protected]>
Co-developed-by Zhang Yi <[email protected]>
Signed-off-by Zhang Yi <[email protected]>
Co-developed-by Yang Weijiang <[email protected]>
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/mmu.c | 50 +++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 11 ++++++++ 3 files changed, 62 insertions(+)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -790,6 +790,7 @@ struct kvm_lpage_info {struct kvm_arch_memory_slot {struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES ]; +u32 *subpage_wp_info;struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES -1 ];unsigned short *gfn_track[KVM_PAGE_TRACK_MAX ]; };
arch/x86/kvm/mmu.c +50 -0
56 lines not on this page · raw message
arch/x86/kvm/x86.c +11 -0
@@ -9317,6 +9317,17 @@ void kvm_arch_destroy_vm(struct kvm *kvm)kvm_hv_destroy_vm (kvm); } +void kvm_subpage_free_memslot (struct kvm_memory_slot *free, +struct kvm_memory_slot *dont) +{ + +if (!dont || free->arch.subpage_wp_info != + dont->arch.subpage_wp_info) { +kvfree (free->arch.subpage_wp_info); + free->arch.subpage_wp_info =NULL ; + } +} +void kvm_arch_free_memslot (struct kvm *kvm,struct kvm_memory_slot *free,struct kvm_memory_slot *dont) {
Signature · 1 line
2.17.2
PATCH v4 5/9 KVM: VMX: Add init/set/get functions for SPP
In reply to Yang Weijianginit_spp() must be called before {get, set}_subpage functions, it creates subpage access bitmaps for memory pages and issues a KVM request to setup SPPT root pages.
kvm_mmu_set_subpages() is to enable SPP bit in EPT leaf page and setup corresponding SPPT entries. The mmu_lock
is held before above operation. If it's called in EPT fault and SPPT mis-config induced handler, mmu_lock is acquired outside the function, otherwise, it's acquired inside it.
kvm_mmu_get_subpages() is used to query access bitmap for protected page, it's also used in EPT fault handler to check whether the fault EPT page is SPP protected as well.
Co-developed-by He Chen <[email protected]>
Signed-off-by He Chen <[email protected]>
Co-developed-by Zhang Yi <[email protected]>
Signed-off-by Zhang Yi <[email protected]>
Co-developed-by Yang Weijiang <[email protected]>
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/include/asm/kvm_host.h | 18 ++++ arch/x86/include/asm/vmx.h | 2 + arch/x86/kvm/mmu.c | 160 ++++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.c | 48 ++++++++++ arch/x86/kvm/x86.c | 40 ++++++++ include/linux/kvm_host.h | 4 +- include/uapi/linux/kvm.h | 9 ++ 7 files changed, 280 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +18 -0
@@ -398,8 +398,13 @@ struct kvm_mmu {void (*invlpg)(struct kvm_vcpu *vcpu,gva_t gva,hpa_t root_hpa);void (*update_pte)(struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp,u64 *spte,const void *pte); +int (*get_subpages)(struct kvm *kvm,struct kvm_subpage *spp_info); +int (*set_subpages)(struct kvm *kvm,struct kvm_subpage *spp_info); +int (*init_spp)(struct kvm *kvm); +hpa_t root_hpa;gpa_t root_cr3; +hpa_t sppt_root;union kvm_mmu_role mmu_role;u8 root_level;u8 shadow_root_level;
@@ -927,6 +932,8 @@ struct kvm_arch {bool guest_can_read_msr_platform_info;bool exception_payload_enabled; + +bool spp_active; };struct kvm_vm_stat {
@@ -1199,6 +1206,11 @@ struct kvm_x86_ops {uint16_t (*nested_get_evmcs_version)(struct kvm_vcpu *vcpu);bool (*need_emulation_on_page_fault)(struct kvm_vcpu *vcpu); + +bool (*get_spp_status)(void ); +int (*get_subpages)(struct kvm *kvm,struct kvm_subpage *spp_info); +int (*set_subpages)(struct kvm *kvm,struct kvm_subpage *spp_info); +int (*init_spp)(struct kvm *kvm); };struct kvm_arch_async_pf {
@@ -1417,6 +1429,12 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva);void kvm_mmu_invpcid_gva (struct kvm_vcpu *vcpu,gva_t gva,unsigned long pcid);void kvm_mmu_new_cr3 (struct kvm_vcpu *vcpu,gpa_t new_cr3,bool skip_tlb_flush); +int kvm_mmu_get_subpages (struct kvm *kvm,struct kvm_subpage *spp_info, +bool mmu_locked); +int kvm_mmu_set_subpages (struct kvm *kvm,struct kvm_subpage *spp_info, +bool mmu_locked); +int kvm_mmu_init_spp (struct kvm *kvm); +void kvm_enable_tdp (void );void kvm_disable_tdp (void );
arch/x86/include/asm/vmx.h +2 -0
@@ -224,6 +224,8 @@ enum vmcs_field {XSS_EXIT_BITMAP_HIGH =0x0000202D ,ENCLS_EXITING_BITMAP =0x0000202E ,ENCLS_EXITING_BITMAP_HIGH =0x0000202F , +SPPT_POINTER =0x00002030 , +SPPT_POINTER_HIGH =0x00002031 ,TSC_MULTIPLIER =0x00002032 ,TSC_MULTIPLIER_HIGH =0x00002033 , GUEST_PHYSICAL_ADDRESS = 0x00002400,
arch/x86/kvm/mmu.c +160 -0
184 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.c +48 -0
89 lines not on this page · raw message
arch/x86/kvm/x86.c +40 -0
@@ -4589,6 +4589,44 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,return r; } +int kvm_get_subpages (struct kvm *kvm, +struct kvm_subpage *spp_info) +{ +int ret; + +mutex_lock (&kvm->slots_lock); + ret =kvm_mmu_get_subpages (kvm, spp_info, false); +mutex_unlock (&kvm->slots_lock); + +return ret; +} +EXPORT_SYMBOL_GPL (kvm_get_subpages); + +int kvm_set_subpages (struct kvm *kvm, +struct kvm_subpage *spp_info) +{ +int ret; + +mutex_lock (&kvm->slots_lock); + ret =kvm_mmu_set_subpages (kvm, spp_info, false); +mutex_unlock (&kvm->slots_lock); + +return ret; +} +EXPORT_SYMBOL_GPL (kvm_set_subpages); + +int kvm_init_spp (struct kvm *kvm) +{ +int ret; + +mutex_lock (&kvm->slots_lock); + ret =kvm_mmu_init_spp (kvm); +mutex_unlock (&kvm->slots_lock); + +return ret; +} +EXPORT_SYMBOL_GPL (kvm_init_spp); +long kvm_arch_vm_ioctl (struct file *filp,unsigned int ioctl,unsigned long arg) {
@@ -9349,6 +9387,8 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *free, }kvm_page_track_free_memslot (free, dont); +if (kvm->arch.spp_active) +kvm_subpage_free_memslot (free, dont); } int kvm_arch_create_memslot(struct kvm *kvm, struct kvm_memory_slot *slot,
include/linux/kvm_host.h +3 -1
@@ -849,7 +849,9 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu);struct kvm_mmu_page *kvm_mmu_get_spp_page (struct kvm_vcpu *vcpu,gfn_t gfn,unsigned int level); - +int kvm_get_subpages (struct kvm *kvm,struct kvm_subpage *spp_info); +int kvm_set_subpages (struct kvm *kvm,struct kvm_subpage *spp_info); +int kvm_init_spp (struct kvm *kvm);#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* * All architectures that want to use vzalloc currently also
include/uapi/linux/kvm.h +9 -0
@@ -102,6 +102,15 @@ struct kvm_userspace_memory_region {__u64 userspace_addr;/* start of the userspace allocated memory */ }; +/* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */ +#define SUBPAGE_MAX_BITMAP 64 +struct kvm_subpage { +__u64 base_gfn; +__u64 npages; +/* sub-page write-access bitmap array */ +__u32 access_map[SUBPAGE_MAX_BITMAP ]; +}; +/* * The bit 0 ~ bit 15 of kvm_memory_region::flags are visible for userspace, * other bits are reserved for kvm internal use which are defined in
Signature · 1 line
2.17.2
PATCH v4 6/9 KVM: VMX: Introduce SPP user-space IOCTLs
In reply to Yang WeijiangUser application, e.g., QEMU or VMI, must initialize SPP before gets/sets SPP subpages, the dynamic initialization is to reduce the extra storage cost if the SPP feature is not not used.
Co-developed-by He Chen <[email protected]>
Signed-off-by He Chen <[email protected]>
Co-developed-by Zhang Yi <[email protected]>
Signed-off-by Zhang Yi <[email protected]>
Co-developed-by Yang Weijiang <[email protected]>
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/kvm/x86.c | 90 ++++++++++++++++++++++++++++++++++++++++ include/linux/kvm_host.h | 4 ++ include/uapi/linux/kvm.h | 3 ++ 3 files changed, 97 insertions(+)
arch/x86/kvm/x86.c +90 -0
@@ -4589,6 +4589,23 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,return r; } +static int kvm_vm_ioctl_get_subpages (struct kvm *kvm, +struct kvm_subpage *spp_info) +{ +return kvm_arch_get_subpages (kvm, spp_info); +} + +static int kvm_vm_ioctl_set_subpages (struct kvm *kvm, +struct kvm_subpage *spp_info) +{ +return kvm_arch_set_subpages (kvm, spp_info); +} + +static int kvm_vm_ioctl_init_spp (struct kvm *kvm) +{ +return kvm_arch_init_spp (kvm); +} +int kvm_get_subpages (struct kvm *kvm,struct kvm_subpage *spp_info) {
@@ -4922,8 +4939,55 @@ long kvm_arch_vm_ioctl(struct file *filp,if (copy_from_user (&hvevfd, argp,sizeof (hvevfd))) goto out; r =kvm_vm_ioctl_hv_eventfd (kvm, &hvevfd); + } +case KVM_SUBPAGES_GET_ACCESS : { +struct kvm_subpage spp_info; + +if (!kvm->arch.spp_active) { + r = -ENODEV ; + goto out; + } + + r = -EFAULT ; +if (copy_from_user (&spp_info, argp,sizeof (spp_info))) + goto out; + + r = -EINVAL ; +if (spp_info.npages ==0 || + spp_info.npages >SUBPAGE_MAX_BITMAP ) + goto out; + + r =kvm_vm_ioctl_get_subpages (kvm, &spp_info); +if (copy_to_user (argp, &spp_info,sizeof (spp_info))) { + r = -EFAULT ; + goto out; + } +break ; + } +case KVM_SUBPAGES_SET_ACCESS : { +struct kvm_subpage spp_info; + +if (!kvm->arch.spp_active) { + r = -ENODEV ; + goto out; + } + + r = -EFAULT ; +if (copy_from_user (&spp_info, argp,sizeof (spp_info))) + goto out; + + r = -EINVAL ; +if (spp_info.npages ==0 || + spp_info.npages >SUBPAGE_MAX_BITMAP ) + goto out; + + r =kvm_vm_ioctl_set_subpages (kvm, &spp_info);break ; } +case KVM_INIT_SPP : { + r =kvm_vm_ioctl_init_spp (kvm); +break ; + }default : r = -ENOTTY ; }
@@ -9925,6 +9989,32 @@ int kvm_arch_update_irqfd_routing(struct kvm *kvm, unsigned int host_irq,return kvm_x86_ops->update_pi_irte (kvm, host_irq, guest_irq, set); } +int kvm_arch_get_subpages (struct kvm *kvm, +struct kvm_subpage *spp_info) +{ +if (!kvm_x86_ops->get_subpages) +return -EINVAL ; + +return kvm_x86_ops->get_subpages (kvm, spp_info); +} + +int kvm_arch_set_subpages (struct kvm *kvm, +struct kvm_subpage *spp_info) +{ +if (!kvm_x86_ops->set_subpages) +return -EINVAL ; + +return kvm_x86_ops->set_subpages (kvm, spp_info); +} + +int kvm_arch_init_spp (struct kvm *kvm) +{ +if (!kvm_x86_ops->init_spp) +return -EINVAL ; + +return kvm_x86_ops->init_spp (kvm); +} +bool kvm_vector_hashing_enabled (void ) {return vector_hashing;
include/linux/kvm_host.h +4 -0
@@ -852,6 +852,10 @@ struct kvm_mmu_page *kvm_mmu_get_spp_page(struct kvm_vcpu *vcpu,int kvm_get_subpages (struct kvm *kvm,struct kvm_subpage *spp_info);int kvm_set_subpages (struct kvm *kvm,struct kvm_subpage *spp_info);int kvm_init_spp (struct kvm *kvm); +int kvm_arch_get_subpages (struct kvm *kvm,struct kvm_subpage *spp_info); +int kvm_arch_set_subpages (struct kvm *kvm,struct kvm_subpage *spp_info); +int kvm_arch_init_spp (struct kvm *kvm); +#ifndef __KVM_HAVE_ARCH_VM_ALLOC/* * All architectures that want to use vzalloc currently also
include/uapi/linux/kvm.h +3 -0
@@ -1246,6 +1246,9 @@ struct kvm_vfio_spapr_tce { struct kvm_userspace_memory_region)#define KVM_SET_TSS_ADDR _IO(KVMIO, 0x47)#define KVM_SET_IDENTITY_MAP_ADDR _IOW(KVMIO, 0x48, __u64) +#define KVM_SUBPAGES_GET_ACCESS _IOR(KVMIO, 0x49, __u64) +#define KVM_SUBPAGES_SET_ACCESS _IOW(KVMIO, 0x4a, __u64) +#define KVM_INIT_SPP _IOW(KVMIO, 0x4b, __u64)/* enable ucontrol for s390 */ struct kvm_s390_ucas_mapping {
Signature · 1 line
2.17.2
PATCH v4 7/9 KVM: VMX: Handle SPP induced vmexit and page fault
In reply to Yang WeijiangIf write to subpage is not allowed, EPT violation is generated, it's propagated to QEMU or VMI to handle.
If the target page is SPP protected, however SPPT missing is encoutered while traversing with gfn, vmexit is generated so that KVM can handle the issue. Any SPPT misconfig will be propagated to QEMU or VMI.
A SPP specific bit(11) is added to exit_qualification and a new exit reason(66) is introduced for SPP.
Co-developed-by He Chen <[email protected]>
Signed-off-by He Chen <[email protected]>
Co-developed-by Zhang Yi <[email protected]>
Signed-off-by Zhang Yi <[email protected]>
Co-developed-by Yang Weijiang <[email protected]>
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/include/asm/vmx.h | 7 ++++ arch/x86/include/uapi/asm/vmx.h | 2 + arch/x86/kvm/mmu.c | 17 ++++++++ arch/x86/kvm/vmx/vmx.c | 71 +++++++++++++++++++++++++++++++++ include/uapi/linux/kvm.h | 5 +++ 5 files changed, 102 insertions(+)
arch/x86/include/asm/vmx.h +7 -0
@@ -547,6 +547,13 @@ struct vmx_msr_entry {#define EPT_VIOLATION_EXECUTABLE (1 << EPT_VIOLATION_EXECUTABLE_BIT)#define EPT_VIOLATION_GVA_TRANSLATED (1 << EPT_VIOLATION_GVA_TRANSLATED_BIT) +/* +* Exit Qualifications for SPPT-Induced vmexits +*/ +#define SPPT_INDUCED_EXIT_TYPE_BIT 11 +#define SPPT_INDUCED_EXIT_TYPE (1 << SPPT_INDUCED_EXIT_TYPE_BIT) +#define SPPT_INTR_INFO_UNBLOCK_NMI INTR_INFO_UNBLOCK_NMI +/* * VM-instruction error numbers */
arch/x86/include/uapi/asm/vmx.h +2 -0
@@ -85,6 +85,7 @@#define EXIT_REASON_PML_FULL 62#define EXIT_REASON_XSAVES 63#define EXIT_REASON_XRSTORS 64 +#define EXIT_REASON_SPP 66#define VMX_EXIT_REASONS \ { EXIT_REASON_EXCEPTION_NMI, "EXCEPTION_NMI" }, \
@@ -141,6 +142,7 @@ { EXIT_REASON_ENCLS, "ENCLS" }, \ { EXIT_REASON_RDSEED, "RDSEED" }, \ { EXIT_REASON_PML_FULL, "PML_FULL" }, \ + { EXIT_REASON_SPP, "SPP" }, \ { EXIT_REASON_XSAVES, "XSAVES" }, \ { EXIT_REASON_XRSTORS, "XRSTORS" }
arch/x86/kvm/mmu.c +17 -0
@@ -3602,6 +3602,19 @@ static bool fast_page_fault(struct kvm_vcpu *vcpu, gva_t gva, int level,if ((error_code &PFERR_WRITE_MASK ) &&spte_can_locklessly_be_made_writable (spte)) { +/* +* Record write protect fault caused by +* Sub-page Protection, let VMI decide +* the next step. +*/ +if (spte &PT_SPP_MASK ) { + fault_handled = true; + vcpu->run->exit_reason =KVM_EXIT_SPP ; + vcpu->run->spp.addr = gva; +kvm_skip_emulated_instruction (vcpu); +break ; + } + new_spte |=PT_WRITABLE_MASK ;/*
@@ -5786,6 +5799,10 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t cr2, u64 error_code, r = vcpu->arch.mmu->page_fault (vcpu, cr2,lower_32_bits (error_code), false); + +if (vcpu->run->exit_reason ==KVM_EXIT_SPP ) +return 0 ; +WARN_ON (r ==RET_PF_INVALID ); }
arch/x86/kvm/vmx/vmx.c +71 -0
83 lines not on this page · raw message
include/uapi/linux/kvm.h +5 -0
@@ -244,6 +244,7 @@ struct kvm_hyperv_exit {#define KVM_EXIT_S390_STSI 25#define KVM_EXIT_IOAPIC_EOI 26#define KVM_EXIT_HYPERV 27 +#define KVM_EXIT_SPP 28/* For KVM_EXIT_INTERNAL_ERROR */ /* Emulate instruction failed. */
@@ -399,6 +400,10 @@ struct kvm_run {struct {__u8 vector; } eoi; +/* KVM_EXIT_SPP */ +struct { +__u64 addr; + } spp;/* KVM_EXIT_HYPERV */ struct kvm_hyperv_exit hyperv;/* Fix the size of the union. */
Signature · 1 line
2.17.2
PATCH v4 8/9 KVM: MMU: Enable Lazy mode SPPT setup
In reply to Yang WeijiangIf SPP subpages are set while the physical page are not available in EPT leaf entry, the mapping is first stored in SPP access bitmap buffer. SPPT setup is deferred to access to the protected page, in EPT page fault handler, the SPPT enries are set up.
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/kvm/mmu.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/x86/kvm/mmu.c +24 -0
@@ -4304,6 +4304,26 @@ check_hugepage_cache_consistency(struct kvm_vcpu *vcpu, gfn_t gfn, int level)return kvm_mtrr_check_gfn_range_consistency (vcpu, gfn, page_num); } +static int kvm_enable_spp_protection (struct kvm *kvm,u64 gfn) +{ +struct kvm_subpage spp_info = {0 }; +struct kvm_memory_slot *slot; + + slot =gfn_to_memslot (kvm, gfn); +if (!slot) +return -EFAULT ; + + spp_info.base_gfn = gfn; + spp_info.npages =1 ; + +if (kvm_mmu_get_subpages (kvm, &spp_info, true) <0 ) +return -EFAULT ; + +if (spp_info.access_map[0 ] !=FULL_SPP_ACCESS ) +kvm_mmu_set_subpages (kvm, &spp_info, true); + +return 0 ; +}static int tdp_page_fault (struct kvm_vcpu *vcpu,gva_t gpa,u32 error_code,bool prefault) {
@@ -4355,6 +4375,10 @@ static int tdp_page_fault(struct kvm_vcpu *vcpu, gva_t gpa, u32 error_code,if (likely (!force_pt_level))transparent_hugepage_adjust (vcpu, &gfn, &pfn, &level); r =__direct_map (vcpu, write, map_writable, level, gfn, pfn, prefault); + +if (vcpu->kvm->arch.spp_active && level ==PT_PAGE_TABLE_LEVEL ) +kvm_enable_spp_protection (vcpu->kvm, gfn); +spin_unlock (&vcpu->kvm->mmu_lock);return r;
Signature · 1 line
2.17.2
PATCH v4 9/9 KVM: MMU: Handle host memory remapping and reclaim
In reply to Yang WeijiangHost page swapping/migration may change the translation in EPT leaf entry, if the target page is SPP protected, re-enable SPP protection in MMU notifier. If SPPT shadow page is reclaimed, the level1 pages don't have rmap to clear.
Signed-off-by Yang Weijiang <[email protected]>
--- arch/x86/kvm/mmu.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/x86/kvm/mmu.c +22 -0
@@ -1926,6 +1926,24 @@ static int kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head, new_spte &= ~PT_WRITABLE_MASK ; new_spte &= ~SPTE_HOST_WRITEABLE ; +/* +* if it's EPT leaf entry and the physical page is +* SPP protected, then re-enable SPP protection for +* the page. +*/ +if (kvm->arch.spp_active && + level ==PT_PAGE_TABLE_LEVEL ) { +struct kvm_subpage spp_info = {0 }; +int i; + + spp_info.base_gfn = gfn; + spp_info.npages =1 ; + i =kvm_mmu_get_subpages (kvm, &spp_info, true); +if (i ==1 && + spp_info.access_map[0 ] !=FULL_SPP_ACCESS ) + new_spte |=PT_SPP_MASK ; + } + new_spte =mark_spte_for_access_track (new_spte);mmu_spte_clear_track_bits (sptep);
@@ -2809,6 +2827,10 @@ static bool mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, pte = *spte;if (is_shadow_present_pte (pte)) {if (is_last_spte (pte, sp->role.level)) { +/* SPPT leaf entries don't have rmaps*/ +if (sp->role.level ==PT_PAGE_TABLE_LEVEL && +is_spp_spte (sp)) +return true;drop_spte (kvm, spte);if (is_large_pte (pte)) --kvm->stat.lpages;
Signature · 1 line
2.17.2