Linear Address Masking (LAM) KVM Enabling
v8 cover Linear Address Masking (LAM) KVM Enabling
===Feature Introduction===
Linear-address masking (LAM) [1], modifies the checking that is applied to *64-bit* linear addresses, allowing software to use of the untranslated address bits for metadata and masks the metadata bits before using them as linear addresses to access memory.
When the feature is virtualized and exposed to guest, it can be used for efficient address sanitizers (ASAN) implementation and for optimizations in JITs and virtual machines.
Regarding which pointer bits are masked and can be used for metadata, LAM has 2 modes:
- LAM_48: metadata bits 62:48, i.e. LAM width of 15.
- LAM_57: metadata bits 62:57, i.e. LAM width of 6.
* For user pointers:
CR3.LAM_U57 = CR3.LAM_U48 = 0, LAM is off; CR3.LAM_U57 = 1, LAM57 is active;
CR3.LAM_U57 = 0 and CR3.LAM_U48 = 1, LAM48 is active.
* For supervisor pointers:
CR4.LAM_SUP =0, LAM is off;
CR4.LAM_SUP =1 with 5-level paging mode, LAM57 is active; CR4.LAM_SUP =1 with 4-level paging mode, LAM48 is active.
The modified LAM canonicality check:
* LAM_S48 : [ 1 ][ metadata ][ 1 ]
63 47
* LAM_U48 : [ 0 ][ metadata ][ 0 ]
63 47
* LAM_S57 : [ 1 ][ metadata ][ 1 ]
63 56
* LAM_U57 + 5-lvl paging : [ 0 ][ metadata ][ 0 ]
63 56
* LAM_U57 + 4-lvl paging : [ 0 ][ metadata ][ 0...0 ]
63 56..47
Note:
1. LAM applies to only data address, not to instructions.
2. LAM identification of an address as user or supervisor is based solely on the value of pointer bit 63 and does not depend on the CPL.
3. LAM doesn't apply to the writes to control registers or MSRs.
4. LAM masking applies before paging, so the faulting linear address in CR2 doesn't contain the metadata.
5 The guest linear address saved in VMCS doesn't contain metadata.
6. For user mode address, it is possible that 5-level paging and LAM_U48 are both set, in this case, the effective usable linear address width is 48. (Currently, only LAM_U57 is enabled in Linux kernel. [2])
===LAM KVM Design===
LAM KVM enabling includes the following parts:
- Feature Enumeration
LAM feature is enumerated by CPUID.7.1:EAX.LAM[bit 26].
If hardware supports LAM and host doesn't disable it explicitly (e.g. via clearcpuid), LAM feature will be exposed to user VMM.
- CR4 Virtualization
LAM uses CR4.LAM_SUP (bit 28) to configure LAM on supervisor pointers. Add support to allow guests to set the new CR4 control bit for guests to enable LAM on supervisor pointers.
- CR3 Virtualization
LAM uses CR3.LAM_U48 (bit 62) and CR3.LAM_U57 (bit 61) to configure LAM on user pointers.
Add support to allow guests to set two new CR3 non-address control bits for guests to enable LAM on user pointers.
- Modified Canonicality Check and Metadata Mask
When LAM is enabled, 64-bit linear address may be tagged with metadata. Linear address should be checked for modified canonicality and untagged in instruction emulations and VMExit handlers when LAM is applicable.
LAM inside nested guest is supported by this patch series. LAM inside SGX enclave mode is NOT supported by this patch series.
LAM QEMU patch:
https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg08036.html
LAM kvm-unit-tests patch:
https://lore.kernel.org/kvm/[email protected]/
===Test===
1. Add test cases in kvm-unit-test [3] for LAM, including LAM_SUP and LAM_{U57,U48}. For supervisor pointers, the test covers CR4 LAM_SUP bits toggle, Memory/MMIO access with tagged pointer, and some special instructions (INVLPG, INVPCID, INVVPID), INVVPID cases also used to cover VMX instruction VMExit path. For uer pointers, the test covers CR3 LAM bits toggle, Memory/MMIO access with tagged pointer.
MMIO cases are used to trigger instruction emulation path.
Run the unit test with both LAM feature on/off (i.e. including negative cases). Run the unit test in L1 guest with both LAM feature on/off.
2. Run Kernel LAM kselftests [2] in guest, with both EPT=Y/N.
3. Launch a nested guest.
All tests have passed in Simics environment.
[1] Intel ISE https://cdrdv2.intel.com/v1/dl/getContent/671368
Chapter Linear Address Masking (LAM)
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/kvm/[email protected]/
---
Changelog
v7 --> v8:
- Rebase the patch series on Linux kernel v6.4-rc1, which has all dependencies of this patch series.
- Add a cleanup patch to consolidate flags for __linearize().
- Some changes in commit message of patch 2.
- Rewrite the commit message of patch 3 (Thanks Kai Huang for the suggestion). Opportunistically use GENMASK_ULL() to define __PT_BASE_ADDR_MASK. Opportunistically use kvm_vcpu_is_legal_cr3() to check CR3 in SVM nested code, to provide a clear distinction b/t CR3 and GPA checks.
- Change interface untag_addr() to optional version to avoid adding a dummy version in SVM in patch 4.
- Verified and confirmed that LAM doesn't apply to the address in descriptor of invvpid, drop the untag action for it in patch 5.
v6 --> v7:
- Changes to CR3 virtualization when EPT off
* Leave LAM bits in root.pgd to force a new root for a CR3+LAM combination. (Sean)
* Perform GFN calculation from guest CR3/PGD generically by extracting the maximal base address mask. (Sean)
- Remove de-reference of ctxt->vcpu in the emulator. (Sean)
- Fix a bug in v6, which hardcoded "write" to "false" by mistake in linearize(). (Chao Gao)
- Add Chao Gao's reviewed-by in Patch 5.
- Add Xuelian Guo's tested-by in the patch set.
- Separate cleanup patches from the patch set.
v5 --> v6:
Add Patch 2 to fix the check of 64-bit mode.
Add untag_addr() to kvm_x86_ops to hide vendor specific code.
Simplify the LAM canonicality check per Chao's suggestion.
Add cr3_ctrl_bits to kvm_vcpu_arch to simplify cr3 invalidation/extract/mask (Chao Gao) Extend the patchset scope to include nested virtualization and SGX ENCLS handling.
- Add X86_CR4_LAM_SUP in cr4_fixed1_update for nested vmx. (Chao Gao)
- Add SGX ENCLS VMExit handling
- Add VMX instruction VMExit handling
More descriptions in cover letter.
Add Chao's reviewed-by on Patch 4.
Add more test cases in kvm-unit-test.
v4 --> v5:
Reorder and melt patches surround CR3.LAM bits into Patch 3 of this version.
Revise Patch 1's subject and description
Drop Patch 3
Use kvm_read_cr4_bits() instead of kvm_read_cr4()
Fix: No need to untag addr when write to msr, it should be legacy canonical check Rename kvm_is_valid_cr3() --> kvm_vcpu_is_valid_cr3(), and update some call sites of kvm_vcpu_is_valid_cr3() to use kvm_is_valid_cr3().
Other refactors and Miscs.
v3 --> v4:
Drop unrelated Patch 1 in v3 (Binbin, Sean, Xiaoyao)
Intercept CR4.LAM_SUP instead of pass through to guest (Sean) Just filter out CR3.LAM_{U48, U57}, instead of all reserved high bits (Sean, Yuan)
Use existing __canonical_address() helper instead write a new one (Weijiang) Add LAM handling in KVM emulation (Yu, Yuan)
Add Jingqi's reviwed-by on Patch 7
Rebased to Kirill's latest code, which is 6.2-rc1 base.
v2 --> v3:
As LAM Kernel patches are in tip tree now, rebase to it.
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/
v1 --> v2:
1. Fixes i386-allyesconfig build error on get_pgd(), where CR3_HIGH_RSVD_MASK isn't applicable.
(Reported-by: kernel test robot <[email protected]>)
2. In kvm_set_cr3(), be conservative on skip tlb flush when only LAM bits toggles. (Kirill)
Binbin Wu (3):
KVM: x86: Consolidate flags for __linearize()
KVM: x86: Introduce untag_addr() in kvm_x86_ops
KVM: x86: Untag address when LAM applicable
Robert Hoo (3):
KVM: x86: Virtualize CR4.LAM_SUP
KVM: x86: Virtualize CR3.LAM_{U48,U57}
KVM: x86: Expose LAM feature to userspace VMM
arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 10 ++++- arch/x86/kvm/cpuid.c | 2 +- arch/x86/kvm/cpuid.h | 5 +++ arch/x86/kvm/emulate.c | 30 +++++++++++---- arch/x86/kvm/kvm_emulate.h | 7 ++++ arch/x86/kvm/mmu.h | 5 +++ arch/x86/kvm/mmu/mmu.c | 8 +++- arch/x86/kvm/mmu/mmu_internal.h | 1 + arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/mmu/spte.h | 2 +- arch/x86/kvm/svm/nested.c | 4 +- arch/x86/kvm/vmx/nested.c | 6 ++- arch/x86/kvm/vmx/sgx.c | 1 + arch/x86/kvm/vmx/vmx.c | 59 +++++++++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.h | 2 + arch/x86/kvm/x86.c | 11 +++++- arch/x86/kvm/x86.h | 2 + 18 files changed, 140 insertions(+), 19 deletions(-)
base-commit: ac9a78681b921877518763ba0e89202254349d1b
Signature · 1 line
2.25.1
PATCH v8 1/6 KVM: x86: Consolidate flags for __linearize()
Define a 32-bit parameter and consolidate the two bools into it.
__linearize() has two bool parameters write and fetch. And new flag will be needed to support new feature (e.g. LAM needs a flag to skip address untag under some conditions).
No functional change intended.
In the follow-up patches, the new parameter will be extended for LAM.
--- arch/x86/kvm/emulate.c | 19 +++++++++++++------ arch/x86/kvm/kvm_emulate.h | 4 ++++ 2 files changed, 17 insertions(+), 6 deletions(-)
arch/x86/kvm/emulate.c +13 -6
@@ -687,8 +687,8 @@ static unsigned insn_alignment(struct x86_emulate_ctxt *ctxt, unsigned size) static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt,struct segmented_address addr,unsigned *max_size,unsigned size, -bool write,bool fetch, -enum x86emul_mode mode,ulong *linear) +u32 flags,enum x86emul_mode mode, +ulong *linear) {struct desc_struct desc;bool usable;
@@ -696,6 +696,8 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt,u32 lim;u16 sel;u8 va_bits; +bool fetch = !!(flags &X86EMUL_F_FETCH ); +bool write = !!(flags &X86EMUL_F_WRITE ); la =seg_base (ctxt, addr.seg) + addr.ea; *max_size =0 ;
@@ -757,7 +759,11 @@ static int linearize(struct x86_emulate_ctxt *ctxt, ulong *linear) {unsigned max_size; -return __linearize (ctxt, addr, &max_size, size, write, false, +u32 flags =0 ; + +if (write) + flags |=X86EMUL_F_WRITE ; +return __linearize (ctxt, addr, &max_size, size, flags, ctxt->mode, linear); }
@@ -768,10 +774,11 @@ static inline int assign_eip(struct x86_emulate_ctxt *ctxt, ulong dst)unsigned max_size;struct segmented_address addr = { .seg =VCPU_SREG_CS , .ea = dst }; +u32 flags =X86EMUL_F_FETCH ;if (ctxt->op_bytes !=sizeof (unsigned long )) addr.ea = dst & ((1UL << (ctxt->op_bytes <<3 )) -1 ); - rc =__linearize (ctxt, addr, &max_size,1 , false, true, ctxt->mode, &linear); + rc =__linearize (ctxt, addr, &max_size,1 , flags, ctxt->mode, &linear);if (rc ==X86EMUL_CONTINUE ) ctxt->_eip = addr.ea;return rc;
@@ -896,6 +903,7 @@ static int __do_insn_fetch_bytes(struct x86_emulate_ctxt *ctxt, int op_size)int cur_size = ctxt->fetch.end - ctxt->fetch.data;struct segmented_address addr = { .seg =VCPU_SREG_CS , .ea = ctxt->eip + cur_size }; +u32 flags =X86EMUL_F_FETCH ;/* * We do not know exactly how many bytes will be needed, and
@@ -907,8 +915,7 @@ static int __do_insn_fetch_bytes(struct x86_emulate_ctxt *ctxt, int op_size)* boundary check itself. Instead, we use max_size to check * against op_size. */ - rc =__linearize (ctxt, addr, &max_size,0 , false, true, ctxt->mode, - &linear); + rc =__linearize (ctxt, addr, &max_size,0 , flags, ctxt->mode, &linear);if (unlikely (rc !=X86EMUL_CONTINUE ))return rc;
arch/x86/kvm/kvm_emulate.h +4 -0
@@ -88,6 +88,10 @@ struct x86_instruction_info {#define X86EMUL_IO_NEEDED 5/* IO is needed to complete emulation */ #define X86EMUL_INTERCEPTED 6/* Intercepted by nested VMCB/VMCS */ +/* x86-specific emulation flags */ +#define X86EMUL_F_FETCH BIT(0) +#define X86EMUL_F_WRITE BIT(1) +struct x86_emulate_ops {void (*vm_bugged)(struct x86_emulate_ctxt *ctxt);/*
Signature · 1 line
2.25.1
PATCH v8 2/6 KVM: x86: Virtualize CR4.LAM_SUP
In reply to Binbin WuFrom: Robert Hoo <[email protected]>
Add support to allow guests to set the new CR4 control bit for guests to enable the new Intel CPU feature Linear Address Masking (LAM) on supervisor pointers.
LAM modifies the checking that is applied to 64-bit linear addresses, allowing software to use of the untranslated address bits for metadata and masks the metadata bits before using them as linear addresses to access memory. LAM uses CR4.LAM_SUP (bit 28) to configure LAM for supervisor pointers. LAM also changes VMENTER to allow the bit to be set in VMCS's HOST_CR4 and GUEST_CR4 for virtualization.
Move CR4.LAM_SUP out of CR4_RESERVED_BITS and its reservation depends on vcpu supporting LAM feature or not. Leave the bit intercepted to avoid vmread every time when KVM fetches its value, with the expectation that guest won't toggle the bit frequently.
Set CR4.LAM_SUP bit in the emulated IA32_VMX_CR4_FIXED1 MSR for guests to allow guests to enable LAM for supervisor pointers in nested VMX operation.
Hardware is not required to do TLB flush when CR4.LAM_SUP toggled, KVM doesn't need to emulate TLB flush based on it.
There's no other features/vmx_exec_controls connection, no other code needed in {kvm,vmx}_set_cr4().
--- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.h | 2 ++ 3 files changed, 7 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +2 -1
@@ -125,7 +125,8 @@ | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ - | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP)) + | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP \ + | X86_CR4_LAM_SUP))#define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR)
arch/x86/kvm/vmx/vmx.c +3 -0
@@ -7603,6 +7603,9 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu)cr4_fixed1_update (X86_CR4_UMIP , ecx,feature_bit (UMIP ));cr4_fixed1_update (X86_CR4_LA57 , ecx,feature_bit (LA57 )); + entry =kvm_find_cpuid_entry_index (vcpu,0x7 ,1 ); +cr4_fixed1_update (X86_CR4_LAM_SUP , eax,feature_bit (LAM )); +#undef cr4_fixed1_update }
arch/x86/kvm/x86.h +2 -0
@@ -529,6 +529,8 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); __reserved_bits |=X86_CR4_VMXE ; \if (!__cpu_has (__c,X86_FEATURE_PCID )) \ __reserved_bits |=X86_CR4_PCIDE ; \ +if (!__cpu_has (__c,X86_FEATURE_LAM )) \ + __reserved_bits |=X86_CR4_LAM_SUP ; \ __reserved_bits; \ })
Signature · 1 line
2.25.1
PATCH v8 3/6 KVM: x86: Virtualize CR3.LAM_{U48,U57}
In reply to Binbin WuFrom: Robert Hoo <[email protected]>
Add support to allow guests to set two new CR3 non-address control bits for guests to enable the new Intel CPU feature Linear Address Masking (LAM) on user pointers.
LAM modifies the checking that is applied to 64-bit linear addresses, allowing software to use of the untranslated address bits for metadata and masks the metadata bits before using them as linear addresses to access memory. LAM uses two new CR3 non-address bits LAM_U48 (bit 62) and AM_U57 (bit 61) to configure LAM for user pointers. LAM also changes VMENTER to allow both bits to be set in VMCS's HOST_CR3 and GUEST_CR3 for virtualization.
When EPT is on, CR3 is not trapped by KVM and it's up to the guest to set any of the two LAM control bits. However, when EPT is off, the actual CR3 used by the guest is generated from the shadow MMU root which is different from the CR3 that is *set* by the guest, and KVM needs to manually apply any active control bits to VMCS's GUEST_CR3 based on the cached CR3 *seen* by the guest.
KVM manually checks guest's CR3 to make sure it points to a valid guest physical address (i.e. to support smaller MAXPHYSADDR in the guest). Extend this check to allow the two LAM control bits to be set. And to make such check generic, introduce a new field 'cr3_ctrl_bits' to vcpu to record all feature control bits that are allowed to be set by the guest. After check, non-address bits of guest CR3 will be stripped off to extract guest physical address.
In case of nested, for a guest which supports LAM, both VMCS12's HOST_CR3 and GUEST_CR3 are allowed to have the new LAM control bits set, i.e. when L0 enters L1 to emulate a VMEXIT from L2 to L1 or when L0 enters L2 directly. KVM also manually checks VMCS12's HOST_CR3 and GUEST_CR3 being valid physical address. Extend such check to allow the new LAM control bits too.
Note, LAM doesn't have a global control bit to turn on/off LAM completely, but purely depends on hardware's CPUID to determine it can be enabled or not. That means, when EPT is on, even when KVM doesn't expose LAM to guest, the guest can still set LAM control bits in CR3 w/o causing problem. This is an unfortunate virtualization hole. KVM could choose to intercept CR3 in this case and inject fault but this would hurt performance when running a normal VM w/o LAM support. This is undesirable. Just choose to let the guest do such illegal thing as the worst case is guest being killed when KVM eventually find out such illegal behaviour and that is the guest to blame.
Opportunistically use GENMASK_ULL() to define __PT_BASE_ADDR_MASK. Opportunistically use kvm_vcpu_is_legal_cr3() to check CR3 in SVM nested code, to provide a clear distinction b/t CR3 and GPA checks.
--- arch/x86/include/asm/kvm_host.h | 5 +++++ arch/x86/kvm/cpuid.h | 5 +++++ arch/x86/kvm/mmu.h | 5 +++++ arch/x86/kvm/mmu/mmu.c | 8 +++++++- arch/x86/kvm/mmu/mmu_internal.h | 1 + arch/x86/kvm/mmu/paging_tmpl.h | 3 ++- arch/x86/kvm/mmu/spte.h | 2 +- arch/x86/kvm/svm/nested.c | 4 ++-- arch/x86/kvm/vmx/nested.c | 4 ++-- arch/x86/kvm/vmx/vmx.c | 6 +++++- arch/x86/kvm/x86.c | 4 ++-- 11 files changed, 37 insertions(+), 10 deletions(-)
arch/x86/include/asm/kvm_host.h +5 -0
@@ -727,6 +727,11 @@ struct kvm_vcpu_arch {unsigned long cr0_guest_owned_bits;unsigned long cr2;unsigned long cr3; +/* +* CR3 non-address feature control bits. +* Guest CR3 may contain any of those bits at runtime. +*/ +u64 cr3_ctrl_bits;unsigned long cr4;unsigned long cr4_guest_owned_bits;unsigned long cr4_guest_rsvd_bits;
arch/x86/kvm/cpuid.h +5 -0
@@ -42,6 +42,11 @@ static inline int cpuid_maxphyaddr(struct kvm_vcpu *vcpu)return vcpu->arch.maxphyaddr; } +static inline bool kvm_vcpu_is_legal_cr3 (struct kvm_vcpu *vcpu,unsigned long cr3) +{ +return !((cr3 & vcpu->arch.reserved_gpa_bits) & ~vcpu->arch.cr3_ctrl_bits); +} +static inline bool kvm_vcpu_is_legal_gpa (struct kvm_vcpu *vcpu,gpa_t gpa) {return !(gpa & vcpu->arch.reserved_gpa_bits);
arch/x86/kvm/mmu.h +5 -0
@@ -144,6 +144,11 @@ static inline unsigned long kvm_get_active_pcid(struct kvm_vcpu *vcpu)return kvm_get_pcid (vcpu,kvm_read_cr3 (vcpu)); } +static inline u64 kvm_get_active_cr3_ctrl_bits (struct kvm_vcpu *vcpu) +{ +return kvm_read_cr3 (vcpu) & vcpu->arch.cr3_ctrl_bits; +} +static inline void kvm_mmu_load_pgd (struct kvm_vcpu *vcpu) {u64 root_hpa = vcpu->arch.mmu->root.hpa;
arch/x86/kvm/mmu/mmu.c +7 -1
@@ -3812,7 +3812,13 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)hpa_t root; root_pgd =kvm_mmu_get_guest_pgd (vcpu, mmu); - root_gfn = root_pgd >>PAGE_SHIFT ; +/* +* Guest PGD can be CR3 or EPTP (for nested EPT case). CR3 may contain +* additional control bits (e.g. LAM control bits). To be generic, +* unconditionally strip non-address bits when computing the GFN since +* the guest PGD has already been checked for validity. +*/ + root_gfn = (root_pgd & __PT_BASE_ADDR_MASK) >>PAGE_SHIFT ;if (mmu_check_root (vcpu, root_gfn))return 1 ;
arch/x86/kvm/mmu/mmu_internal.h +1 -0
@@ -21,6 +21,7 @@ extern bool dbg;#endif /* Page table builder macros common to shadow (host) PTEs and guest PTEs. */ +#define __PT_BASE_ADDR_MASK GENMASK_ULL(51, 12)#define __PT_LEVEL_SHIFT (level, bits_per_level) \ (PAGE_SHIFT + ((level) - 1) * (bits_per_level))#define __PT_INDEX (address, level, bits_per_level) \
arch/x86/kvm/mmu/paging_tmpl.h +2 -1
@@ -62,7 +62,7 @@#endif /* Common logic, but per-type values. These also need to be undefined. */ -#define PT_BASE_ADDR_MASK ((pt_element_t)(((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1))) +#define PT_BASE_ADDR_MASK ((pt_element_t)__PT_BASE_ADDR_MASK)#define PT_LVL_ADDR_MASK (lvl) __PT_LVL_ADDR_MASK(PT_BASE_ADDR_MASK, lvl, PT_LEVEL_BITS)#define PT_LVL_OFFSET_MASK (lvl) __PT_LVL_OFFSET_MASK(PT_BASE_ADDR_MASK, lvl, PT_LEVEL_BITS)#define PT_INDEX (addr, lvl) __PT_INDEX(addr, lvl, PT_LEVEL_BITS)
@@ -324,6 +324,7 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker,trace_kvm_mmu_pagetable_walk (addr, access); retry_walk: walker->level = mmu->cpu_role.base.level; +/* gpte_to_gfn() will strip non-address bits. */ pte =kvm_mmu_get_guest_pgd (vcpu, mmu); have_ad =PT_HAVE_ACCESSED_DIRTY (mmu);
arch/x86/kvm/mmu/spte.h +1 -1
@@ -36,7 +36,7 @@ static_assert(SPTE_TDP_AD_ENABLED == 0);#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK #define SPTE_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1))#else -#define SPTE_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1)) +#define SPTE_BASE_ADDR_MASK __PT_BASE_ADDR_MASK#endif #define SPTE_PERM_MASK (PT_PRESENT_MASK | PT_WRITABLE_MASK | shadow_user_mask \
arch/x86/kvm/svm/nested.c +2 -2
@@ -311,7 +311,7 @@ static bool __nested_vmcb_check_save(struct kvm_vcpu *vcpu,if ((save->efer &EFER_LME ) && (save->cr0 &X86_CR0_PG )) {if (CC (!(save->cr4 &X86_CR4_PAE )) ||CC (!(save->cr0 &X86_CR0_PE )) || -CC (kvm_vcpu_is_illegal_gpa (vcpu, save->cr3))) +CC (!kvm_vcpu_is_legal_cr3 (vcpu, save->cr3)))return false; }
@@ -520,7 +520,7 @@ static void nested_svm_transition_tlb_flush(struct kvm_vcpu *vcpu)static int nested_svm_load_cr3 (struct kvm_vcpu *vcpu,unsigned long cr3,bool nested_npt,bool reload_pdptrs) { -if (CC (kvm_vcpu_is_illegal_gpa (vcpu, cr3))) +if (CC (!kvm_vcpu_is_legal_cr3 (vcpu, cr3)))return -EINVAL ; if (reload_pdptrs && !nested_npt && is_pae_paging(vcpu) &&
arch/x86/kvm/vmx/nested.c +2 -2
@@ -1085,7 +1085,7 @@ static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_ept, bool reload_pdptrs, enum vm_entry_failure_code *entry_failure_code) { - if (CC(kvm_vcpu_is_illegal_gpa(vcpu, cr3))) { + if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) { *entry_failure_code =ENTRY_FAIL_DEFAULT ;return -EINVAL ; }
@@ -2913,7 +2913,7 @@ static int nested_vmx_check_host_state(struct kvm_vcpu *vcpu,if (CC (!nested_host_cr0_valid (vcpu, vmcs12->host_cr0)) ||CC (!nested_host_cr4_valid (vcpu, vmcs12->host_cr4)) || -CC (kvm_vcpu_is_illegal_gpa (vcpu, vmcs12->host_cr3))) +CC (!kvm_vcpu_is_legal_cr3 (vcpu, vmcs12->host_cr3)))return -EINVAL ; if (CC(is_noncanonical_address(vmcs12->host_ia32_sysenter_esp, vcpu)) ||
arch/x86/kvm/vmx/vmx.c +5 -1
@@ -3361,7 +3361,8 @@ static void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, update_guest_cr3 = false;vmx_ept_load_pdptrs (vcpu); }else { - guest_cr3 = root_hpa |kvm_get_active_pcid (vcpu); + guest_cr3 = root_hpa |kvm_get_active_pcid (vcpu) | +kvm_get_active_cr3_ctrl_bits (vcpu); }if (update_guest_cr3)
@@ -7743,6 +7744,9 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vmx->msr_ia32_feature_control_valid_bits &= ~FEAT_CTL_SGX_LC_ENABLED ; +if (guest_cpuid_has (vcpu,X86_FEATURE_LAM )) + vcpu->arch.cr3_ctrl_bits |=X86_CR3_LAM_U48 |X86_CR3_LAM_U57 ; +/* Refresh #PF interception to account for MAXPHYADDR changes. */ vmx_update_exception_bitmap (vcpu); }
arch/x86/kvm/x86.c +2 -2
@@ -1275,7 +1275,7 @@ int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3)* stuff CR3, e.g. for RSM emulation, and there is no guarantee that * the current vCPU mode is accurate. */ -if (kvm_vcpu_is_illegal_gpa (vcpu, cr3)) +if (!kvm_vcpu_is_legal_cr3 (vcpu, cr3))return 1 ;if (is_pae_paging (vcpu) && !load_pdptrs (vcpu, cr3))
@@ -11443,7 +11443,7 @@ static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs)*/ if (!(sregs->cr4 &X86_CR4_PAE ) || !(sregs->efer &EFER_LMA ))return false; -if (kvm_vcpu_is_illegal_gpa (vcpu, sregs->cr3)) +if (!kvm_vcpu_is_legal_cr3 (vcpu, sregs->cr3))return false; }else {/*
Signature · 1 line
2.25.1
PATCH v8 4/6 KVM: x86: Introduce untag_addr() in kvm_x86_ops
In reply to Binbin WuIntroduce a new optional interface untag_addr() to kvm_x86_ops to untag the metadata from linear address. Implement LAM version in VMX.
When enabled feature like Intel Linear Address Masking or AMD Upper Address Ignore, linear address may be tagged with metadata. Linear address should be checked for modified canonicality and untagged in instruction emulations or VMExit handlers if LAM or UAI is applicable.
Introduce untag_addr() to kvm_x86_ops to hide the vendor specific code. Pass the 'flags' to avoid distinguishing processor vendor in common emulator path for the cases whose untag policies are different in the future.
For VMX, LAM version is implemented.
LAM has a modified canonical check when applicable:
* LAM_S48 : [ 1 ][ metadata ][ 1 ]
63 47
* LAM_U48 : [ 0 ][ metadata ][ 0 ]
63 47
* LAM_S57 : [ 1 ][ metadata ][ 1 ]
63 56
* LAM_U57 + 5-lvl paging : [ 0 ][ metadata ][ 0 ]
63 56
* LAM_U57 + 4-lvl paging : [ 0 ][ metadata ][ 0...0 ]
63 56..47
LAM identification of an address as user or supervisor is based solely on the value of pointer bit 63.
If LAM is applicable to certain address, untag the metadata bits by sign-extending the value of bit 47 (LAM48) or bit 56 (LAM57). Later the untagged address will do legacy canonical check. So that LAM canonical check and mask can be covered by "untag + legacy canonical check".
--- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/kvm_emulate.h | 1 + arch/x86/kvm/vmx/vmx.c | 50 ++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.h | 2 ++ 5 files changed, 56 insertions(+)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
@@ -52,6 +52,7 @@ KVM_X86_OP(cache_reg) KVM_X86_OP(get_rflags) KVM_X86_OP(set_rflags) KVM_X86_OP(get_if_flag) +KVM_X86_OP_OPTIONAL(untag_addr) KVM_X86_OP(flush_tlb_all) KVM_X86_OP(flush_tlb_current) KVM_X86_OP_OPTIONAL(flush_remote_tlbs)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1588,6 +1588,8 @@ struct kvm_x86_ops {void (*set_rflags)(struct kvm_vcpu *vcpu,unsigned long rflags);bool (*get_if_flag)(struct kvm_vcpu *vcpu); +void (*untag_addr)(struct kvm_vcpu *vcpu,u64 *la,u32 flags); +void (*flush_tlb_all)(struct kvm_vcpu *vcpu);void (*flush_tlb_current)(struct kvm_vcpu *vcpu);int (*flush_remote_tlbs)(struct kvm *kvm);
arch/x86/kvm/kvm_emulate.h +1 -0
@@ -91,6 +91,7 @@ struct x86_instruction_info {/* x86-specific emulation flags */ #define X86EMUL_F_FETCH BIT(0)#define X86EMUL_F_WRITE BIT(1) +#define X86EMUL_F_SKIPLAM BIT(2)struct x86_emulate_ops {void (*vm_bugged)(struct x86_emulate_ctxt *ctxt);
arch/x86/kvm/vmx/vmx.c +50 -0
62 lines not on this page · raw message
arch/x86/kvm/vmx/vmx.h +2 -0
@@ -433,6 +433,8 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type);u64 vmx_get_l2_tsc_offset (struct kvm_vcpu *vcpu);u64 vmx_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu); +void vmx_untag_addr (struct kvm_vcpu *vcpu,u64 *la,u32 flags); +static inline void vmx_set_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type,bool value) {
Signature · 1 line
2.25.1
PATCH v8 5/6 KVM: x86: Untag address when LAM applicable
In reply to Binbin WuUntag address for 64-bit memory/MMIO operand in instruction emulations and VMExit handlers when LAM is applicable.
For instruction emulation, untag address in __linearize() before canonical check. LAM doesn't apply to addresses used for instruction fetches or to those that specify the targets of jump and call instructions, use X86EMUL_F_SKIPLAM to skip LAM untag.
For VMExit handlers related to 64-bit linear address:
- Cases need to untag address
Operand(s) of VMX instructions and INVPCID.
Operand(s) of SGX ENCLS.
- Cases LAM doesn't apply to
Operand of INVLPG.
Linear address in INVPCID descriptor (no change needed).
Linear address in INVVPID descriptor (it has been confirmed, although it is not called out in LAM spec, no change needed).
Note:
LAM doesn't apply to the writes to control registers or MSRs. LAM masking applies before paging, so the faulting linear address in CR2 doesn't contain the metadata.
The guest linear address saved in VMCS doesn't contain metadata.
--- arch/x86/kvm/emulate.c | 15 ++++++++++++--- arch/x86/kvm/kvm_emulate.h | 2 ++ arch/x86/kvm/vmx/nested.c | 2 ++ arch/x86/kvm/vmx/sgx.c | 1 + arch/x86/kvm/x86.c | 7 +++++++ 5 files changed, 24 insertions(+), 3 deletions(-)
arch/x86/kvm/emulate.c +12 -3
@@ -703,6 +703,7 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt, *max_size =0 ;switch (mode) {case X86EMUL_MODE_PROT64 : + ctxt->ops->untag_addr (ctxt, (u64 *)&la, flags); *linear = la; va_bits =ctxt_virt_addr_bits (ctxt);if (!__is_canonical_address (la, va_bits))
@@ -774,7 +775,11 @@ static inline int assign_eip(struct x86_emulate_ctxt *ctxt, ulong dst)unsigned max_size;struct segmented_address addr = { .seg =VCPU_SREG_CS , .ea = dst }; -u32 flags =X86EMUL_F_FETCH ; +/* +* LAM doesn't apply to addresses that specify the targets of jump and +* call instructions. +*/ +u32 flags =X86EMUL_F_FETCH |X86EMUL_F_SKIPLAM ;if (ctxt->op_bytes !=sizeof (unsigned long )) addr.ea = dst & ((1UL << (ctxt->op_bytes <<3 )) -1 );
@@ -903,7 +908,8 @@ static int __do_insn_fetch_bytes(struct x86_emulate_ctxt *ctxt, int op_size)int cur_size = ctxt->fetch.end - ctxt->fetch.data;struct segmented_address addr = { .seg =VCPU_SREG_CS , .ea = ctxt->eip + cur_size }; -u32 flags =X86EMUL_F_FETCH ; +/* LAM doesn't apply to addresses used for instruction fetches */ +u32 flags =X86EMUL_F_FETCH |X86EMUL_F_SKIPLAM ;/* * We do not know exactly how many bytes will be needed, and
@@ -3448,8 +3454,11 @@ static int em_invlpg(struct x86_emulate_ctxt *ctxt) {int rc;ulong linear; +unsigned max_size; - rc =linearize (ctxt, ctxt->src.addr.mem,1 , false, &linear); +/* LAM doesn't apply to invlpg */ + rc =__linearize (ctxt, ctxt->src.addr.mem, &max_size,1 , +X86EMUL_F_SKIPLAM , ctxt->mode, &linear);if (rc ==X86EMUL_CONTINUE ) ctxt->ops->invlpg (ctxt, linear);/* Disable writeback. */
arch/x86/kvm/kvm_emulate.h +2 -0
@@ -230,6 +230,8 @@ struct x86_emulate_ops {int (*leave_smm)(struct x86_emulate_ctxt *ctxt);void (*triple_fault)(struct x86_emulate_ctxt *ctxt);int (*set_xcr)(struct x86_emulate_ctxt *ctxt,u32 index,u64 xcr); + +void (*untag_addr)(struct x86_emulate_ctxt *ctxt,u64 *addr,u32 flags); };/* Type, address-of, and value of an instruction's operand. */
arch/x86/kvm/vmx/nested.c +2 -0
@@ -4981,6 +4981,7 @@ int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsigned long exit_qualification,else *ret = off; +vmx_untag_addr (vcpu, (u64 *)ret,0 );/* Long mode: #GP(0)/#SS(0) if the memory address is in a * non-canonical form. This is the only check on the memory * destination for long mode!
@@ -5798,6 +5799,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) vpid02 =nested_get_vpid02 (vcpu);switch (type) {case VMX_VPID_EXTENT_INDIVIDUAL_ADDR : +/* LAM doesn't apply to the address in descriptor of invvpid */ if (!operand.vpid ||is_noncanonical_address (operand.gla, vcpu)) return nested_vmx_fail(vcpu,
arch/x86/kvm/vmx/sgx.c +1 -0
@@ -37,6 +37,7 @@ static int sgx_get_encls_gva(struct kvm_vcpu *vcpu, unsigned long offset,if (!IS_ALIGNED (*gva, alignment)) { fault = true; }else if (likely (is_64_bit_mode (vcpu))) { +vmx_untag_addr (vcpu, (u64 *)gva,0 ); fault =is_noncanonical_address (*gva, vcpu); }else { *gva &=0xffffffff ;
arch/x86/kvm/x86.c +7 -0
@@ -8283,6 +8283,11 @@ static void emulator_vm_bugged(struct x86_emulate_ctxt *ctxt)kvm_vm_bugged (kvm); } +static void emulator_untag_addr (struct x86_emulate_ctxt *ctxt,u64 *addr,u32 flags) +{ +static_call (kvm_x86_untag_addr)(emul_to_vcpu (ctxt), addr, flags); +} +static const struct x86_emulate_ops emulate_ops = { .vm_bugged = emulator_vm_bugged, .read_gpr = emulator_read_gpr,
@@ -8328,6 +8333,7 @@ static const struct x86_emulate_ops emulate_ops = { .leave_smm = emulator_leave_smm, .triple_fault = emulator_triple_fault, .set_xcr = emulator_set_xcr, + .untag_addr = emulator_untag_addr, };static void toggle_interruptibility (struct kvm_vcpu *vcpu,u32 mask)
@@ -13354,6 +13360,7 @@ int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva)switch (type) {case INVPCID_TYPE_INDIV_ADDR : +/* LAM doesn't apply to the address in descriptor of invpcid */ if ((!pcid_enabled && (operand.pcid !=0 )) ||is_noncanonical_address (operand.gla, vcpu)) {kvm_inject_gp (vcpu,0 );
Signature · 1 line
2.25.1
PATCH v8 6/6 KVM: x86: Expose LAM feature to userspace VMM
In reply to Binbin WuFrom: Robert Hoo <[email protected]>
LAM feature is enumerated by CPUID.7.1:EAX.LAM[bit 26]. Expose the feature to userspace as the final step after the following supports:
- CR4.LAM_SUP virtualization
- CR3.LAM_U48 and CR3.LAM_U57 virtualization
- Check and untag 64-bit linear address when LAM applies in instruction emulations and VMExit handlers.
--- arch/x86/kvm/cpuid.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/cpuid.c +1 -1
@@ -664,7 +664,7 @@ void kvm_set_cpu_caps(void)kvm_cpu_cap_mask (CPUID_7_1_EAX ,F (AVX_VNNI ) |F (AVX512_BF16 ) |F (CMPCCXADD ) |F (FZRM ) |F (FSRS ) |F (FSRC ) | -F (AMX_FP16 ) |F (AVX_IFMA ) +F (AMX_FP16 ) |F (AVX_IFMA ) |F (LAM ) ); kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,
Signature · 1 line
2.25.1
Re: [PATCH v8 1/6] KVM: x86: Consolidate flags for __linearize()
In reply to Binbin WuQuoted 11 lines · Binbin Wu
Define a 32-bit parameter and consolidate the two bools into it.
__linearize()has two bool parameters write and fetch. And new flag will be needed to support new feature (e.g. LAM needs a flag to skip address untag under some conditions).No functional change intended.
In the follow-up patches, the new parameter will be extended for LAM.
Signed-off-by Binbin Wu <[email protected]>
one nit below
Quoted 30 lines
---
arch/x86/kvm/emulate.c| 19 +++++++++++++------arch/x86/kvm/kvm_emulate.h| 4 ++++2 files changed, 17 insertions(+), 6 deletions(-)arch/x86/kvm/emulate.c +4
-2@@ -687,8 +687,8 @@ static unsigned insn_alignment(struct x86_emulate_ctxt *ctxt, unsigned size) static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt,struct segmented_address addr,unsigned *max_size,unsigned size, -bool write,bool fetch, -enum x86emul_mode mode,ulong *linear) +u32 flags,enum x86emul_mode mode, +ulong *linear) {struct desc_struct desc;bool usable;@@ -696,6 +696,8 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt,u32 lim;u16 sel;u8 va_bits; +bool fetch = !!(flags &X86EMUL_F_FETCH ); +bool write = !!(flags &X86EMUL_F_WRITE ); la =seg_base (ctxt, addr.seg) + addr.ea; *max_size =0 ;
arch/x86/kvm/emulate.c static int linearize(struct x86_emulate_ctxt *ctxt, · line 765
Quoted 8 lines
arch/x86/kvm/emulate.c +4
-1@@ -757,7 +759,11 @@ static int linearize(struct x86_emulate_ctxt *ctxt, ulong *linear) {unsigned max_size; - return __linearize(ctxt, addr, &max_size, size, write, false, +u32 flags =0 ; + +if (write) + flags |=X86EMUL_F_WRITE ;
this can be more dense:
u32 flags = write ? X86EMUL_F_WRITE : 0;
Re: [PATCH v8 3/6] KVM: x86: Virtualize CR3.LAM_{U48,U57}
In reply to Binbin Wuarch/x86/kvm/vmx/vmx.c static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) · line 7748
Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +2
-0@@ -7743,6 +7744,9 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vmx->msr_ia32_feature_control_valid_bits &= ~FEAT_CTL_SGX_LC_ENABLED ; +if (guest_cpuid_has (vcpu,X86_FEATURE_LAM )) + vcpu->arch.cr3_ctrl_bits |=X86_CR3_LAM_U48 |X86_CR3_LAM_U57 ;
This function can be called multiple times. We need to clear LAM bits if LAM isn't exposed to the guest, i.e.,
else vcpu->arch.cr3_ctrl_bits &= ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57);
With this fixed,
Re: [PATCH v8 3/6] KVM: x86: Virtualize CR3.LAM_{U48,U57}
In reply to Binbin WuQuoted 52 lines · Binbin Wu
From: Robert Hoo <[email protected]>
Add support to allow guests to set two new CR3 non-address control bits for guests to enable the new Intel CPU feature Linear Address Masking (LAM) on user pointers.
LAM modifies the checking that is applied to 64-bit linear addresses, allowing software to use of the untranslated address bits for metadata and masks the metadata bits before using them as linear addresses to access memory. LAM uses two new CR3 non-address bits
LAM_U48(bit 62) andAM_U57(bit 61) to configure LAM for user pointers. LAM also changes VMENTER to allow both bits to be set in VMCS'sHOST_CR3andGUEST_CR3for virtualization.When EPT is on, CR3 is not trapped by KVM and it's up to the guest to set any of the two LAM control bits. However, when EPT is off, the actual CR3 used by the guest is generated from the shadow MMU root which is different from the CR3 that is *set* by the guest, and KVM needs to manually apply any active control bits to VMCS's
GUEST_CR3based on the cached CR3 *seen* by the guest.KVM manually checks guest's CR3 to make sure it points to a valid guest physical address (i.e. to support smaller MAXPHYSADDR in the guest). Extend this check to allow the two LAM control bits to be set. And to make such check generic, introduce a new field '
cr3_ctrl_bits' to vcpu to record all feature control bits that are allowed to be set by the guest. After check, non-address bits of guest CR3 will be stripped off to extract guest physical address.In case of nested, for a guest which supports LAM, both VMCS12's
HOST_CR3andGUEST_CR3are allowed to have the new LAM control bits set, i.e. when L0 enters L1 to emulate a VMEXIT from L2 to L1 or when L0 enters L2 directly. KVM also manually checks VMCS12'sHOST_CR3andGUEST_CR3being valid physical address. Extend such check to allow the new LAM control bits too.Note, LAM doesn't have a global control bit to turn on/off LAM completely, but purely depends on hardware's CPUID to determine it can be enabled or not. That means, when EPT is on, even when KVM doesn't expose LAM to guest, the guest can still set LAM control bits in CR3 w/o causing problem. This is an unfortunate virtualization hole. KVM could choose to intercept CR3 in this case and inject fault but this would hurt performance when running a normal VM w/o LAM support. This is undesirable. Just choose to let the guest do such illegal thing as the worst case is guest being killed when KVM eventually find out such illegal behaviour and that is the guest to blame.
Opportunistically use
GENMASK_ULL()to define__PT_BASE_ADDR_MASK. Opportunistically usekvm_vcpu_is_legal_cr3()to check CR3 in SVM nested code, to provide a clear distinction b/t CR3 and GPA checks.Suggested-by Sean Christopherson <[email protected]>Signed-off-by Robert Hoo <[email protected]>Co-developed-by Binbin Wu <[email protected]>Signed-off-by Binbin Wu <[email protected]>Tested-by Xuelian Guo <[email protected]>
[...]
Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +0
-0@@ -3812,7 +3812,13 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu)hpa_t root;
arch/x86/kvm/mmu/mmu.c static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) · line 3821
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +7
-1root_pgd =kvm_mmu_get_guest_pgd (vcpu, mmu); - root_gfn = root_pgd >>PAGE_SHIFT ; +/* +* Guest PGD can be CR3 or EPTP (for nested EPT case). CR3 may contain +* additional control bits (e.g. LAM control bits). To be generic, +* unconditionally strip non-address bits when computing the GFN since +* the guest PGD has already been checked for validity. +*/ + root_gfn = (root_pgd & __PT_BASE_ADDR_MASK) >>PAGE_SHIFT ;
Looks it's better to mask off non-address bits based on guest's invalid physical address bits, for instance, based on vcpu->arch.resereved_gpa_bits. But looking at the old discussion it appears Sean suggested this way:
https://lore.kernel.org/all/[email protected]/
So anyway:
Re: [PATCH v8 1/6] KVM: x86: Consolidate flags for __linearize()
In reply to Binbin WuQuoted 1 line · Binbin Wu
Define a 32-bit parameter and consolidate the two bools into it.
Technically, per C standard I don't think you can "define" a parameter of a function, but can only "declare".
Quoted 4 lines
__linearize()has two bool parameters write and fetch. And new flag will be needed to support new feature (e.g. LAM needs a flag to skip address untag under some conditions).
Since this is the first patch to mention LAM in this series, it would be better to use the full name Intel Linear Address Masking (LAM).
Quoted 4 lines
No functional change intended.
In the follow-up patches, the new parameter will be extended for LAM.
A duplicated sentence to me. Perhaps you can just remove it.
Some changelog material FYI:
Consolidate two bool parameters (write/fetch) of __linearize() into a 'u32 flag' parameter to make the function be more concise and future extendable, i.e. to support Intel Linear Address Masking (LAM), which allows high non-address bits of linear address to be used as metadata.
Define two flags to replace the two bools. A new flag will be added to to support LAM to skip masking off metadata bits of linear address
under
some conditions.
No functional change intended.
Quoted 2 lines
Signed-off-by Binbin Wu <[email protected]>
Anyway:
Re: [PATCH v8 1/6] KVM: x86: Consolidate flags for __linearize()
In reply to Chao GaoQuoted 56 lines · Chao Gao
Quoted 11 lines · Binbin Wu
Define a 32-bit parameter and consolidate the two bools into it.
__linearize()has two bool parameters write and fetch. And new flag will be needed to support new feature (e.g. LAM needs a flag to skip address untag under some conditions).No functional change intended.
In the follow-up patches, the new parameter will be extended for LAM.
Signed-off-by Binbin Wu <[email protected]>Reviewed-by Chao Gao <[email protected]>one nit below
Quoted 30 lines
---
arch/x86/kvm/emulate.c| 19 +++++++++++++------arch/x86/kvm/kvm_emulate.h| 4 ++++2 files changed, 17 insertions(+), 6 deletions(-)arch/x86/kvm/emulate.c +4
-2@@ -687,8 +687,8 @@ static unsigned insn_alignment(struct x86_emulate_ctxt *ctxt, unsigned size) static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt,struct segmented_address addr,unsigned *max_size,unsigned size, -bool write,bool fetch, -enum x86emul_mode mode,ulong *linear) +u32 flags,enum x86emul_mode mode, +ulong *linear) {struct desc_struct desc;bool usable;@@ -696,6 +696,8 @@ static __always_inline int __linearize(struct x86_emulate_ctxt *ctxt,u32 lim;u16 sel;u8 va_bits; +bool fetch = !!(flags &X86EMUL_F_FETCH ); +bool write = !!(flags &X86EMUL_F_WRITE ); la =seg_base (ctxt, addr.seg) + addr.ea; *max_size =0 ;arch/x86/kvm/emulate.c static int linearize(struct x86_emulate_ctxt *ctxt, · line 765
Quoted 8 lines
arch/x86/kvm/emulate.c +4
-1@@ -757,7 +759,11 @@ static int linearize(struct x86_emulate_ctxt *ctxt, ulong *linear) {unsigned max_size; - return __linearize(ctxt, addr, &max_size, size, write, false, +u32 flags =0 ; + +if (write) + flags |=X86EMUL_F_WRITE ;this can be more dense:
Quoted 1 line
u32 flags = write ? X86EMUL_F_WRITE : 0;
Thanks, will update it.
Re: [PATCH v8 3/6] KVM: x86: Virtualize CR3.LAM_{U48,U57}
In reply to Chao GaoQuoted 11 lines · Chao Gao
arch/x86/kvm/vmx/vmx.c static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) · line 7748
Quoted 6 lines
arch/x86/kvm/vmx/vmx.c +2
-0@@ -7743,6 +7744,9 @@ static void vmx_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) vmx->msr_ia32_feature_control_valid_bits &= ~FEAT_CTL_SGX_LC_ENABLED ; +if (guest_cpuid_has (vcpu,X86_FEATURE_LAM )) + vcpu->arch.cr3_ctrl_bits |=X86_CR3_LAM_U48 |X86_CR3_LAM_U57 ;This function can be called multiple times. We need to clear LAM bits if LAM isn't exposed to the guest, i.e.,
else vcpu->arch.cr3_ctrl_bits &= ~(X86_CR3_LAM_U48 | X86_CR3_LAM_U57);
Nice catch, will fix it, thanks.
Quoted 4 lines
With this fixed,
Reviewed-by Chao Gao <[email protected]>
Re: [PATCH v8 1/6] KVM: x86: Consolidate flags for __linearize()
In reply to Huang, KaiQuoted 23 lines · Kai
Quoted 1 line · Binbin Wu
Define a 32-bit parameter and consolidate the two bools into it.
Technically, per C standard I don't think you can "define" a parameter of a function, but can only "declare".
Quoted 3 lines
__linearize()has two bool parameters write and fetch. And new flag will be needed to support new feature (e.g. LAM needs a flag to skip address untag under some conditions).Since this is the first patch to mention LAM in this series, it would be better to use the full name Intel Linear Address Masking (LAM).
Quoted 3 lines
No functional change intended.
In the follow-up patches, the new parameter will be extended for LAM.
A duplicated sentence to me. Perhaps you can just remove it.
Some changelog material FYI:
Consolidate two bool parameters (write/fetch) of
__linearize()into a 'u32 flag' parameter to make the function be more concise and future extendable, i.e. to support Intel Linear Address Masking (LAM), which allows high non-address bits of linear address to be used as metadata.
Quoted 6 lines
Define two flags to replace the two bools. A new flag will be added to to support LAM to skip masking off metadata bits of linear address
under
some conditions.No functional change intended.
Thanks, will update it.
Quoted 5 lines
Quoted 2 lines
Signed-off-by Binbin Wu <[email protected]>Anyway:
Acked-by Kai Huang <[email protected]>
Re: [PATCH v8 4/6] KVM: x86: Introduce untag_addr() in kvm_x86_ops
In reply to Binbin WuQuoted 25 lines · Binbin Wu
Introduce a new optional interface
untag_addr()tokvm_x86_opsto untag the metadata from linear address. Implement LAM version in VMX.When enabled feature like Intel Linear Address Masking or AMD Upper Address Ignore, linear address may be tagged with metadata. Linear address should be checked for modified canonicality and untagged in instruction emulations or VMExit handlers if LAM or UAI is applicable.
Introduce
untag_addr()tokvm_x86_opsto hide the vendor specific code. Pass the 'flags' to avoid distinguishing processor vendor in common emulator path for the cases whose untag policies are different in the future.For VMX, LAM version is implemented.
LAM has a modified canonical check when applicable:
*LAM_S48: [ 1 ][ metadata ][ 1 ]
63 47
*LAM_U48: [ 0 ][ metadata ][ 0 ]
63 47
*LAM_S57: [ 1 ][ metadata ][ 1 ]
63 56
*LAM_U57+ 5-lvl paging : [ 0 ][ metadata ][ 0 ]
63 56
*LAM_U57+ 4-lvl paging : [ 0 ][ metadata ][ 0...0 ]
Quoted 7 lines
63 56..47LAM identification of an address as user or supervisor is based solely on the value of pointer bit 63.
If LAM is applicable to certain address, untag the metadata bits by sign-extending the value of bit 47 (LAM48) or bit 56 (LAM57). Later the untagged address will do legacy canonical check. So that LAM canonical check and mask can be covered by "untag + legacy canonical check".
Do you think it is better to remove such details from the changelog and instead document them in comments e.g.,
Quoted 52 lines
Signed-off-by Binbin Wu <[email protected]>Tested-by Xuelian Guo <[email protected]>---
arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/kvm_emulate.h | 1 + arch/x86/kvm/vmx/vmx.c | 50 ++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.h | 2 ++ 5 files changed, 56 insertions(+)arch/x86/include/asm/kvm-x86-ops.h +1
-0@@ -52,6 +52,7 @@ KVM_X86_OP(cache_reg) KVM_X86_OP(get_rflags) KVM_X86_OP(set_rflags) KVM_X86_OP(get_if_flag) +KVM_X86_OP_OPTIONAL(untag_addr) KVM_X86_OP(flush_tlb_all) KVM_X86_OP(flush_tlb_current) KVM_X86_OP_OPTIONAL(flush_remote_tlbs)arch/x86/include/asm/kvm_host.h +2
-0@@ -1588,6 +1588,8 @@ struct kvm_x86_ops {void (*set_rflags)(struct kvm_vcpu *vcpu,unsigned long rflags);bool (*get_if_flag)(struct kvm_vcpu *vcpu); +void (*untag_addr)(struct kvm_vcpu *vcpu,u64 *la,u32 flags); +void (*flush_tlb_all)(struct kvm_vcpu *vcpu);void (*flush_tlb_current)(struct kvm_vcpu *vcpu);int (*flush_remote_tlbs)(struct kvm *kvm);arch/x86/kvm/kvm_emulate.h +1
-0@@ -91,6 +91,7 @@ struct x86_instruction_info {/* x86-specific emulation flags */ #define X86EMUL_F_FETCH BIT(0)#define X86EMUL_F_WRITE BIT(1) +#define X86EMUL_F_SKIPLAM BIT(2)struct x86_emulate_ops {void (*vm_bugged)(struct x86_emulate_ctxt *ctxt);arch/x86/kvm/vmx/vmx.c +0
-0
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8138
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +2
-0@@ -8134,6 +8134,54 @@ static void vmx_vm_destroy(struct kvm *kvm)free_pages ((unsigned long )kvm_vmx->pid_table,vmx_get_pid_table_order (kvm)); } +#define LAM_S57_EN_MASK (X86_CR4_LAM_SUP | X86_CR4_LA57) +static int lam_sign_extend_bit (struct kvm_vcpu *vcpu,bool user)
we can let this function take an address ...
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8141
Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +3
-0+{ +u64 cr3, cr4; +
then it is natural to add the comment -- "LAM identification ...", like
/*
* The LAM identification of a pointer as user or supervisor is
* based solely on the value of pointer bit 63.
*/
if (!(addr >> 63)) {Quoted 14 lines
arch/x86/kvm/vmx/vmx.c +14
-0+if (user) { + cr3 =kvm_read_cr3 (vcpu); +if (cr3 &X86_CR3_LAM_U57 ) +return 56 ; +if (cr3 &X86_CR3_LAM_U48 ) +return 47 ; + }else { + cr4 =kvm_read_cr4_bits (vcpu,LAM_S57_EN_MASK ); +if (cr4 ==LAM_S57_EN_MASK ) +return 56 ; +if (cr4 &X86_CR4_LAM_SUP ) +return 47 ; + } +return -1 ;
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8165
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10
-0+} + +/* +* Only called in 64-bit mode. +* +* Metadata bits are [62:48] in LAM48 and [62:57] in LAM57. Mask metadata in +* pointers by sign-extending the value of bit 47 (LAM48) or 56 (LAM57). +* The resulting address after untagging isn't guaranteed to be canonical. +* Callers should perform the original canonical check and raise #GP/#SS if the +* address is non-canonical.
To document the difference between KVM's emulation of LAM and real hardware behavior:
*
* Note that KVM masks the metadata in addresses, performs the (original)
* canonicality checking and then walks page table. This is slightly
* different from hardware behavior but achieves the same effect.
* Specifically, if LAM is enabled, the processor performs a modified
* canonicality checking where the metadata are ignored instead of
* masked. After the modified canonicality checking, the processor masks
* the metadata before passing addresses for paging translation. */
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8167
Quoted 2 lines
arch/x86/kvm/vmx/vmx.c +2
-0+*/ +void vmx_untag_addr (struct kvm_vcpu *vcpu,u64 *la,u32 flags)
^
nit: "gla" or "gva" is slightly better. It may be hard to immediately connect "la" with "linear address".
or even better, use "gva_t gva"?
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +5
-0+{ +int sign_ext_bit; + +/* +* Check LAM_U48 in cr3_ctrl_bits to avoid guest_cpuid_has().
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8182
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10
-0+* If not set, vCPU doesn't supports LAM. +*/ +if (!(vcpu->arch.cr3_ctrl_bits &X86_CR3_LAM_U48 ) || + (flags &X86EMUL_F_SKIPLAM ) ||WARN_ON_ONCE (!is_64_bit_mode (vcpu))) +return ; + + sign_ext_bit =lam_sign_extend_bit (vcpu, !(*la >>63 )); +if (sign_ext_bit >0 ) + *la = (sign_extend64 (*la, sign_ext_bit) & ~BIT_ULL (63 )) | + (*la &BIT_ULL (63 ));
nit: curly braces are needed.
Overall the patch looks good to me, so
(regardless of whether you choose to make the suggested changes to the changelog/comments)
Quoted 30 lines
arch/x86/kvm/vmx/vmx.c +4
-0+} +static struct kvm_x86_ops vmx_x86_ops__initdata = { .name =KBUILD_MODNAME ,@@ -8182,6 +8230,8 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .set_rflags = vmx_set_rflags, .get_if_flag = vmx_get_if_flag, + .untag_addr = vmx_untag_addr, + .flush_tlb_all = vmx_flush_tlb_all, .flush_tlb_current = vmx_flush_tlb_current, .flush_tlb_gva = vmx_flush_tlb_gva,arch/x86/kvm/vmx/vmx.h +2
-0@@ -433,6 +433,8 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type);u64 vmx_get_l2_tsc_offset (struct kvm_vcpu *vcpu);u64 vmx_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu); +void vmx_untag_addr (struct kvm_vcpu *vcpu,u64 *la,u32 flags); +static inline void vmx_set_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type,bool value) {Signature · 1 line
2.25.1
Re: [PATCH v8 5/6] KVM: x86: Untag address when LAM applicable
In reply to Binbin WuQuoted 24 lines · Binbin Wu
Untag address for 64-bit memory/MMIO operand in instruction emulations and VMExit handlers when LAM is applicable.
For instruction emulation, untag address in
__linearize()before canonical check. LAM doesn't apply to addresses used for instruction fetches or to those that specify the targets of jump and call instructions, useX86EMUL_F_SKIPLAMto skip LAM untag.For VMExit handlers related to 64-bit linear address:
- Cases need to untag address
Operand(s) of VMX instructions and INVPCID.
Operand(s) of SGX ENCLS.
- Cases LAM doesn't apply to
Operand of INVLPG.
Linear address in INVPCID descriptor (no change needed).
Linear address in INVVPID descriptor (it has been confirmed, although it is not called out in LAM spec, no change needed).Note:
LAM doesn't apply to the writes to control registers or MSRs. LAM masking applies before paging, so the faulting linear address in CR2 doesn't contain the metadata.
The guest linear address saved in VMCS doesn't contain metadata.
Quoted 4 lines
Co-developed-by Robert Hoo <[email protected]>Signed-off-by Robert Hoo <[email protected]>Signed-off-by Binbin Wu <[email protected]>Tested-by Xuelian Guo <[email protected]>
arch/x86/kvm/kvm_emulate.h struct x86_emulate_ops { · line 234
Quoted 8 lines
arch/x86/kvm/kvm_emulate.h +2
-0@@ -230,6 +230,8 @@ struct x86_emulate_ops {int (*leave_smm)(struct x86_emulate_ctxt *ctxt);void (*triple_fault)(struct x86_emulate_ctxt *ctxt);int (*set_xcr)(struct x86_emulate_ctxt *ctxt,u32 index,u64 xcr); + +void (*untag_addr)(struct x86_emulate_ctxt *ctxt,u64 *addr,u32 flags);
Nit: using 'gva_t *' as the second parameter would eliminate the need to cast to 'u64 *' repeatedly.
Re: [PATCH v8 4/6] KVM: x86: Introduce untag_addr() in kvm_x86_ops
In reply to Chao GaoQuoted 99 lines · Chao Gao
Quoted 24 lines · Binbin Wu
Introduce a new optional interface
untag_addr()tokvm_x86_opsto untag the metadata from linear address. Implement LAM version in VMX.When enabled feature like Intel Linear Address Masking or AMD Upper Address Ignore, linear address may be tagged with metadata. Linear address should be checked for modified canonicality and untagged in instruction emulations or VMExit handlers if LAM or UAI is applicable.
Introduce
untag_addr()tokvm_x86_opsto hide the vendor specific code. Pass the 'flags' to avoid distinguishing processor vendor in common emulator path for the cases whose untag policies are different in the future.For VMX, LAM version is implemented.
LAM has a modified canonical check when applicable:
*LAM_S48: [ 1 ][ metadata ][ 1 ]
63 47
*LAM_U48: [ 0 ][ metadata ][ 0 ]
63 47
*LAM_S57: [ 1 ][ metadata ][ 1 ]
63 56
*LAM_U57+ 5-lvl paging : [ 0 ][ metadata ][ 0 ]
63 56
*LAM_U57+ 4-lvl paging : [ 0 ][ metadata ][ 0...0 ]Quoted 7 lines
63 56..47LAM identification of an address as user or supervisor is based solely on the value of pointer bit 63.
If LAM is applicable to certain address, untag the metadata bits by sign-extending the value of bit 47 (LAM48) or bit 56 (LAM57). Later the untagged address will do legacy canonical check. So that LAM canonical check and mask can be covered by "untag + legacy canonical check".Do you think it is better to remove such details from the changelog and instead document them in comments e.g.,
Quoted 52 lines
Signed-off-by Binbin Wu <[email protected]>Tested-by Xuelian Guo <[email protected]>---
arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/kvm_emulate.h | 1 + arch/x86/kvm/vmx/vmx.c | 50 ++++++++++++++++++++++++++++++ arch/x86/kvm/vmx/vmx.h | 2 ++ 5 files changed, 56 insertions(+)arch/x86/include/asm/kvm-x86-ops.h +1
-0@@ -52,6 +52,7 @@ KVM_X86_OP(cache_reg) KVM_X86_OP(get_rflags) KVM_X86_OP(set_rflags) KVM_X86_OP(get_if_flag) +KVM_X86_OP_OPTIONAL(untag_addr) KVM_X86_OP(flush_tlb_all) KVM_X86_OP(flush_tlb_current) KVM_X86_OP_OPTIONAL(flush_remote_tlbs)arch/x86/include/asm/kvm_host.h +2
-0@@ -1588,6 +1588,8 @@ struct kvm_x86_ops {void (*set_rflags)(struct kvm_vcpu *vcpu,unsigned long rflags);bool (*get_if_flag)(struct kvm_vcpu *vcpu); +void (*untag_addr)(struct kvm_vcpu *vcpu,u64 *la,u32 flags); +void (*flush_tlb_all)(struct kvm_vcpu *vcpu);void (*flush_tlb_current)(struct kvm_vcpu *vcpu);int (*flush_remote_tlbs)(struct kvm *kvm);arch/x86/kvm/kvm_emulate.h +1
-0@@ -91,6 +91,7 @@ struct x86_instruction_info {/* x86-specific emulation flags */ #define X86EMUL_F_FETCH BIT(0)#define X86EMUL_F_WRITE BIT(1) +#define X86EMUL_F_SKIPLAM BIT(2)struct x86_emulate_ops {void (*vm_bugged)(struct x86_emulate_ctxt *ctxt);arch/x86/kvm/vmx/vmx.c +0
-0arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8138
Quoted 5 lines
arch/x86/kvm/vmx/vmx.c +2
-0@@ -8134,6 +8134,54 @@ static void vmx_vm_destroy(struct kvm *kvm)free_pages ((unsigned long )kvm_vmx->pid_table,vmx_get_pid_table_order (kvm)); } +#define LAM_S57_EN_MASK (X86_CR4_LAM_SUP | X86_CR4_LA57) +static int lam_sign_extend_bit (struct kvm_vcpu *vcpu,bool user)we can let this function take an address ...
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8141
Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +3
-0+{ +u64 cr3, cr4; +then it is natural to add the comment -- "LAM identification ...", like
Quoted 5 lines
/*
* The LAM identification of a pointer as user or supervisor is
* based solely on the value of pointer bit 63.
*/
if (!(addr >> 63)) {OK.
Quoted 28 lines
Quoted 15 lines
arch/x86/kvm/vmx/vmx.c +14
-0+if (user) { + cr3 =kvm_read_cr3 (vcpu); +if (cr3 &X86_CR3_LAM_U57 ) +return 56 ; +if (cr3 &X86_CR3_LAM_U48 ) +return 47 ; + }else { + cr4 =kvm_read_cr4_bits (vcpu,LAM_S57_EN_MASK ); +if (cr4 ==LAM_S57_EN_MASK ) +return 56 ; +if (cr4 &X86_CR4_LAM_SUP ) +return 47 ; + } +return -1 ;arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8165
Quoted 10 lines
arch/x86/kvm/vmx/vmx.c +10
-0+} + +/* +* Only called in 64-bit mode. +* +* Metadata bits are [62:48] in LAM48 and [62:57] in LAM57. Mask metadata in +* pointers by sign-extending the value of bit 47 (LAM48) or 56 (LAM57). +* The resulting address after untagging isn't guaranteed to be canonical. +* Callers should perform the original canonical check and raise #GP/#SS if the +* address is non-canonical. To document the difference between KVM's emulation of LAM and real hardware behavior:
Quoted 9 lines
*
* Note that KVM masks the metadata in addresses, performs the (original)
* canonicality checking and then walks page table. This is slightly
* different from hardware behavior but achieves the same effect.
* Specifically, if LAM is enabled, the processor performs a modified
* canonicality checking where the metadata are ignored instead of
* masked. After the modified canonicality checking, the processor masks
* the metadata before passing addresses for paging translation. */
OK.
Quoted 9 lines
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8167
Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +2
-0+*/ +void vmx_untag_addr (struct kvm_vcpu *vcpu,u64 *la,u32 flags)^
nit: "gla" or "gva" is slightly better. It may be hard to immediately connect "la" with "linear address".or even better, use "
gva_tgva"?
Will update it to use "gva_t gva", thanks.
Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/vmx/vmx.c +8
-0+{ +int sign_ext_bit; + +/* +* Check LAM_U48 in cr3_ctrl_bits to avoid guest_cpuid_has(). +* If not set, vCPU doesn't supports LAM. +*/ + if (!(vcpu->arch.cr3_ctrl_bits & X86_CR3_LAM_U48) ||
Quoted 8 lines
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8182
Quoted 7 lines
arch/x86/kvm/vmx/vmx.c +7
-0+ (flags & X86EMUL_F_SKIPLAM) || WARN_ON_ONCE(!is_64_bit_mode(vcpu))) +return ; + + sign_ext_bit =lam_sign_extend_bit (vcpu, !(*la >>63 )); +if (sign_ext_bit >0 ) + *la = (sign_extend64 (*la, sign_ext_bit) & ~BIT_ULL (63 )) | + (*la &BIT_ULL (63 ));nit: curly braces are needed.
Even if it's only one statement (although splited to two lines), curly braces are needed?
Quoted 38 lines
Overall the patch looks good to me, so
Reviewed-by Chao Gao <[email protected]>(regardless of whether you choose to make the suggested changes to the changelog/comments)
Quoted 30 lines
arch/x86/kvm/vmx/vmx.c +4
-0+} +static struct kvm_x86_ops vmx_x86_ops__initdata = { .name =KBUILD_MODNAME ,@@ -8182,6 +8230,8 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = { .set_rflags = vmx_set_rflags, .get_if_flag = vmx_get_if_flag, + .untag_addr = vmx_untag_addr, + .flush_tlb_all = vmx_flush_tlb_all, .flush_tlb_current = vmx_flush_tlb_current, .flush_tlb_gva = vmx_flush_tlb_gva,arch/x86/kvm/vmx/vmx.h +2
-0@@ -433,6 +433,8 @@ void vmx_enable_intercept_for_msr(struct kvm_vcpu *vcpu, u32 msr, int type);u64 vmx_get_l2_tsc_offset (struct kvm_vcpu *vcpu);u64 vmx_get_l2_tsc_multiplier (struct kvm_vcpu *vcpu); +void vmx_untag_addr (struct kvm_vcpu *vcpu,u64 *la,u32 flags); +static inline void vmx_set_intercept_for_msr (struct kvm_vcpu *vcpu,u32 msr,int type,bool value) {Signature · 1 line
2.25.1
RE: [PATCH v8 1/6] KVM: x86: Consolidate flags for __linearize()
In reply to Binbin WuFrom: Binbin Wu
Quoted 1 line
Sent: 11 May 2023 02:26
...
Quoted 10 lines
Quoted 9 lines
arch/x86/kvm/emulate.c static int linearize(struct x86_emulate_ctxt *ctxt, · line 765
Quoted 6 lines
unsigned max_size;arch/x86/kvm/emulate.c +4
-1- return __linearize(ctxt, addr, &max_size, size, write, false, +u32 flags =0 ; + +if (write) + flags |=X86EMUL_F_WRITE ;this can be more dense:
u32 flags = write ? X86EMUL_F_WRITE : 0;Thanks, will update it.
You can also dispense with the extra local variable and put the ?: into the parameter list.
Even more so with the other calls sites.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v8 4/6] KVM: x86: Introduce untag_addr() in kvm_x86_ops
In reply to Binbin WuQuoted 6 lines · Binbin Wu
Quoted 4 lines
arch/x86/kvm/vmx/vmx.c static void vmx_vm_destroy(struct kvm *kvm) · line 8182
Quoted 3 lines
arch/x86/kvm/vmx/vmx.c +3
-0+if (sign_ext_bit >0 ) + *la = (sign_extend64 (*la, sign_ext_bit) & ~BIT_ULL (63 )) | + (*la &BIT_ULL (63 ));nit: curly braces are needed.
Even if it's only one statement (although splited to two lines), curly braces are needed?
I was under the impression somehow -- braces are needed if the statement is split into two lines. But Sean said [1] curly braces are not needed in this case. Thus, please disregard my comment.
Re: [PATCH v8 2/6] KVM: x86: Virtualize CR4.LAM_SUP
In reply to Binbin WuQuoted 69 lines · Binbin Wu
From: Robert Hoo <[email protected]>
Add support to allow guests to set the new CR4 control bit for guests to enable the new Intel CPU feature Linear Address Masking (LAM) on supervisor pointers.
LAM modifies the checking that is applied to 64-bit linear addresses, allowing software to use of the untranslated address bits for metadata and masks the metadata bits before using them as linear addresses to access memory. LAM uses CR4.LAM_SUP (bit 28) to configure LAM for supervisor pointers. LAM also changes VMENTER to allow the bit to be set in VMCS's
HOST_CR4andGUEST_CR4for virtualization.Move CR4.LAM_SUP out of
CR4_RESERVED_BITSand its reservation depends on vcpu supporting LAM feature or not. Leave the bit intercepted to avoid vmread every time when KVM fetches its value, with the expectation that guest won't toggle the bit frequently.Set CR4.LAM_SUP bit in the emulated
IA32_VMX_CR4_FIXED1MSR for guests to allow guests to enable LAM for supervisor pointers in nested VMX operation.Hardware is not required to do TLB flush when CR4.LAM_SUP toggled, KVM doesn't need to emulate TLB flush based on it.
There's no other features/vmx_exec_controls connection, no other code needed in {kvm,vmx}_set_cr4().Signed-off-by Robert Hoo <[email protected]>Co-developed-by Binbin Wu <[email protected]>Signed-off-by Binbin Wu <[email protected]>Reviewed-by Chao Gao <[email protected]>Tested-by Xuelian Guo <[email protected]>--- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.h | 2 ++ 3 files changed, 7 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +2
-1@@ -125,7 +125,8 @@ | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ - | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP)) + | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP \ + | X86_CR4_LAM_SUP))#define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR)arch/x86/kvm/vmx/vmx.c +3
-0@@ -7603,6 +7603,9 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu)cr4_fixed1_update (X86_CR4_UMIP , ecx,feature_bit (UMIP ));cr4_fixed1_update (X86_CR4_LA57 , ecx,feature_bit (LA57 )); + entry =kvm_find_cpuid_entry_index (vcpu,0x7 ,1 ); +cr4_fixed1_update (X86_CR4_LAM_SUP , eax,feature_bit (LAM )); +#undef cr4_fixed1_update }arch/x86/kvm/x86.h +0
-0
arch/x86/kvm/x86.h bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); · line 536
Quoted 8 lines
arch/x86/kvm/x86.h +2
-0@@ -529,6 +529,8 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); __reserved_bits |=X86_CR4_VMXE ; \if (!__cpu_has (__c,X86_FEATURE_PCID )) \ __reserved_bits |=X86_CR4_PCIDE ; \ +if (!__cpu_has (__c,X86_FEATURE_LAM )) \ + __reserved_bits |=X86_CR4_LAM_SUP ; \ __reserved_bits; \ })
LAM only applies to 64-bit linear address, which means LAM can only be enabled when CPU is in 64-bit mode with either 4-level or 5-level paging enabled.
What's the hardware behaviour if we set CR4.LAM_SUP when CPU isn't in 64-bit mode? And how does VMENTRY check GUEST_CR4.LAM_SUP and 64-bit mode?
Looks they are not clear in the spec you pasted in the cover letter:
https://cdrdv2.intel.com/v1/dl/getContent/671368
Or I am missing something?
Re: [PATCH v8 2/6] KVM: x86: Virtualize CR4.LAM_SUP
Quoted 81 lines · Kai
Quoted 69 lines · Binbin Wu
From: Robert Hoo <[email protected]>
Add support to allow guests to set the new CR4 control bit for guests to enable the new Intel CPU feature Linear Address Masking (LAM) on supervisor pointers.
LAM modifies the checking that is applied to 64-bit linear addresses, allowing software to use of the untranslated address bits for metadata and masks the metadata bits before using them as linear addresses to access memory. LAM uses CR4.LAM_SUP (bit 28) to configure LAM for supervisor pointers. LAM also changes VMENTER to allow the bit to be set in VMCS's
HOST_CR4andGUEST_CR4for virtualization.Move CR4.LAM_SUP out of
CR4_RESERVED_BITSand its reservation depends on vcpu supporting LAM feature or not. Leave the bit intercepted to avoid vmread every time when KVM fetches its value, with the expectation that guest won't toggle the bit frequently.Set CR4.LAM_SUP bit in the emulated
IA32_VMX_CR4_FIXED1MSR for guests to allow guests to enable LAM for supervisor pointers in nested VMX operation.Hardware is not required to do TLB flush when CR4.LAM_SUP toggled, KVM doesn't need to emulate TLB flush based on it.
There's no other features/vmx_exec_controls connection, no other code needed in {kvm,vmx}_set_cr4().Signed-off-by Robert Hoo <[email protected]>Co-developed-by Binbin Wu <[email protected]>Signed-off-by Binbin Wu <[email protected]>Reviewed-by Chao Gao <[email protected]>Tested-by Xuelian Guo <[email protected]>--- arch/x86/include/asm/kvm_host.h | 3 ++- arch/x86/kvm/vmx/vmx.c | 3 +++ arch/x86/kvm/x86.h | 2 ++ 3 files changed, 7 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +2
-1@@ -125,7 +125,8 @@ | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ - | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP)) + | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP \ + | X86_CR4_LAM_SUP))#define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR)arch/x86/kvm/vmx/vmx.c +3
-0@@ -7603,6 +7603,9 @@ static void nested_vmx_cr_fixed1_bits_update(struct kvm_vcpu *vcpu)cr4_fixed1_update (X86_CR4_UMIP , ecx,feature_bit (UMIP ));cr4_fixed1_update (X86_CR4_LA57 , ecx,feature_bit (LA57 )); + entry =kvm_find_cpuid_entry_index (vcpu,0x7 ,1 ); +cr4_fixed1_update (X86_CR4_LAM_SUP , eax,feature_bit (LAM )); +#undef cr4_fixed1_update }arch/x86/kvm/x86.h +0
-0arch/x86/kvm/x86.h bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); · line 536
Quoted 8 lines
arch/x86/kvm/x86.h +2
-0@@ -529,6 +529,8 @@ bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); __reserved_bits |=X86_CR4_VMXE ; \if (!__cpu_has (__c,X86_FEATURE_PCID )) \ __reserved_bits |=X86_CR4_PCIDE ; \ +if (!__cpu_has (__c,X86_FEATURE_LAM )) \ + __reserved_bits |=X86_CR4_LAM_SUP ; \ __reserved_bits; \ })LAM only applies to 64-bit linear address, which means LAM can only be enabled when CPU is in 64-bit mode with either 4-level or 5-level paging enabled.
Quoted 8 lines
What's the hardware behaviour if we set CR4.LAM_SUP when CPU isn't in 64-bit mode? And how does VMENTRY check GUEST_CR4.LAM_SUP and 64-bit mode?
Looks they are not clear in the spec you pasted in the cover letter:
https://cdrdv2.intel.com/v1/dl/getContent/671368
Or I am missing something?
Yes, it is not clearly described in LAM spec.
Had some internal discussions and also did some tests in host,
if the processor supports LAM, CR4.LAM_SUP is allowed to be set even when cpu isn't in 64bit mode.
There was a statement in commit message of the last version, but I missed it in this version. I'll add it back. "CR4.LAM_SUP is allowed to be set even not in 64-bit mode, but it will not
take effect since LAM only applies to 64-bit linear address."
Also, I will try to ask Intel guys if it's possible to update the document.
Re: [PATCH v8 1/6] KVM: x86: Consolidate flags for __linearize()
In reply to David LaightQuoted 11 lines · David Laight
From: Binbin Wu
Quoted 1 line
Sent: 11 May 2023 02:26
...
Quoted 8 lines
Quoted 8 lines
arch/x86/kvm/emulate.c static int linearize(struct x86_emulate_ctxt *ctxt, · line 765
Quoted 6 lines
unsigned max_size;arch/x86/kvm/emulate.c +4
-1- return __linearize(ctxt, addr, &max_size, size, write, false, +u32 flags =0 ; + +if (write) + flags |=X86EMUL_F_WRITE ;this can be more dense:
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
u32 flags = write ? X86EMUL_F_WRITE : 0;Thanks, will update it.
You can also dispense with the extra local variable and put the ?: into the parameter list.
Even more so with the other calls sites.
Thanks, I will check whether they are better to be put in the parameter list directly instead of using a local variable.
Quoted 6 lines
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Re: [PATCH v8 2/6] KVM: x86: Virtualize CR4.LAM_SUP
In reply to Binbin WuQuoted 18 lines
Quoted 5 lines
Quoted 2 lines
LAM only applies to 64-bit linear address, which means LAM can only be enabled when CPU is in 64-bit mode with either 4-level or 5-level paging enabled.
Quoted 8 lines
What's the hardware behaviour if we set CR4.LAM_SUP when CPU isn't in 64-bit mode? And how does VMENTRY check GUEST_CR4.LAM_SUP and 64-bit mode?
Looks they are not clear in the spec you pasted in the cover letter:
https://cdrdv2.intel.com/v1/dl/getContent/671368
Or I am missing something?
Yes, it is not clearly described in LAM spec.
Had some internal discussions and also did some tests in host, if the processor supports LAM, CR4.LAM_SUP is allowed to be set even when cpu isn't in 64bit mode.
Quoted 4 lines
There was a statement in commit message of the last version, but I missed it in this version. I'll add it back.
"CR4.LAM_SUP is allowed to be set even not in 64-bit mode, but it will not take effect since LAM only applies to 64-bit linear address."
Yeah this does help. Please add it back to the changelog.
Quoted 3 lines
Also, I will try to ask Intel guys if it's possible to update the document.
Thanks.
Re: [PATCH v8 6/6] KVM: x86: Expose LAM feature to userspace VMM
In reply to Binbin WuQuoted 24 lines · Binbin Wu
From: Robert Hoo <[email protected]>
LAM feature is enumerated by CPUID.7.1:EAX.LAM[bit 26]. Expose the feature to userspace as the final step after the following supports:
- CR4.LAM_SUP virtualization
- CR3.LAM_U48 and CR3.LAM_U57 virtualization
- Check and untag 64-bit linear address when LAM applies in instruction emulations and VMExit handlers.Signed-off-by Robert Hoo <[email protected]>Signed-off-by Binbin Wu <[email protected]>Reviewed-by Jingqi Liu <[email protected]>Reviewed-by Chao Gao <[email protected]>Tested-by Xuelian Guo <[email protected]>--- arch/x86/kvm/cpuid.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kvm/cpuid.c +0
-0
arch/x86/kvm/cpuid.c void kvm_set_cpu_caps(void) · line 670
Quoted 8 lines
arch/x86/kvm/cpuid.c +1
-1@@ -664,7 +664,7 @@ void kvm_set_cpu_caps(void)kvm_cpu_cap_mask (CPUID_7_1_EAX ,F (AVX_VNNI ) |F (AVX512_BF16 ) |F (CMPCCXADD ) |F (FZRM ) |F (FSRS ) |F (FSRC ) | -F (AMX_FP16 ) |F (AVX_IFMA ) +F (AMX_FP16 ) |F (AVX_IFMA ) |F (LAM ) ); kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,
Per spec LAM supports SGX enclave mode too (chapter 6.8 INTEL SGX INTERACTIONS).
Per my read, whether you can enable LAM57 or LAM48 can be determined by bit 8 and 9 of CPUID.(EAX=12H, ECX=01H):EAX. There are some special rule of whether LAM is applied to certain linear address too, i.e. per my read SECS.BASEADDR isn't subject to LAM in ECREATE.
My first glance to expose LAM to guest's SGX enclave is just as simple as what you did here -- we just need to expose the two new bits to userspace in KVM_GET_SUPPORTED_CPUID.
But I think this can be done in a separate series, especially as the spec doesn't say when LAM is available in CPUID(0x7, 0x1):EAX, it will be always available in CPUID(0x12, 0x1).EAX.
But could you help to check the hardware hehaviour whether they will always appear together in CPUID?
Re: [PATCH v8 6/6] KVM: x86: Expose LAM feature to userspace VMM
Quoted 32 lines · Kai
Quoted 31 lines · Binbin Wu
From: Robert Hoo <[email protected]>
LAM feature is enumerated by CPUID.7.1:EAX.LAM[bit 26]. Expose the feature to userspace as the final step after the following supports:
- CR4.LAM_SUP virtualization
- CR3.LAM_U48 and CR3.LAM_U57 virtualization
- Check and untag 64-bit linear address when LAM applies in instruction emulations and VMExit handlers.Signed-off-by Robert Hoo <[email protected]>Signed-off-by Binbin Wu <[email protected]>Reviewed-by Jingqi Liu <[email protected]>Reviewed-by Chao Gao <[email protected]>Tested-by Xuelian Guo <[email protected]>--- arch/x86/kvm/cpuid.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/kvm/cpuid.c +1
-1@@ -664,7 +664,7 @@ void kvm_set_cpu_caps(void)kvm_cpu_cap_mask (CPUID_7_1_EAX ,F (AVX_VNNI ) |F (AVX512_BF16 ) |F (CMPCCXADD ) |F (FZRM ) |F (FSRS ) |F (FSRC ) | -F (AMX_FP16 ) |F (AVX_IFMA ) +F (AMX_FP16 ) |F (AVX_IFMA ) |F (LAM ) );
Quoted 7 lines
arch/x86/kvm/cpuid.c void kvm_set_cpu_caps(void) · line 670
Quoted 1 line
arch/x86/kvm/cpuid.c +0
-0kvm_cpu_cap_init_kvm_defined(CPUID_7_1_EDX,Per spec LAM supports SGX enclave mode too (chapter 6.8 INTEL SGX INTERACTIONS).
Per my read, whether you can enable LAM57 or LAM48 can be determined by bit 8 and 9 of CPUID.(EAX=12H, ECX=01H):EAX. There are some special rule of whether LAM is applied to certain linear address too, i.e. per my read SECS.BASEADDR isn't subject to LAM in ECREATE.
This is considered when implement patch 5, but you remind me that I should add the materials to describe the handling of SGX system leaf functions in the chagelog.
Quoted 6 lines
My first glance to expose LAM to guest's SGX enclave is just as simple as what you did here -- we just need to expose the two new bits to userspace in
KVM_GET_SUPPORTED_CPUID.But I think this can be done in a separate series,
Yes, support LAM inside SGX enclave is not included in the inital enabling patch series.
Quoted 6 lines
especially as the spec
doesn't say when LAM is available inCPUID(0x7, 0x1):EAX, it will be always available inCPUID(0x12, 0x1).EAX.But could you help to check the hardware hehaviour whether they will always appear together in CPUID?
For this part, I need furtuer investigation.
Will get back to you when I get the answer.
Re: [PATCH v8 2/6] KVM: x86: Virtualize CR4.LAM_SUP
In reply to Huang, KaiQuoted 21 lines · Kai
Quoted 17 lines
Quoted 4 lines
Quoted 1 line
LAM only applies to 64-bit linear address, which means LAM can only be enabled when CPU is in 64-bit mode with either 4-level or 5-level paging enabled.
Quoted 8 lines
What's the hardware behaviour if we set CR4.LAM_SUP when CPU isn't in 64-bit mode? And how does VMENTRY check GUEST_CR4.LAM_SUP and 64-bit mode?
Looks they are not clear in the spec you pasted in the cover letter:
https://cdrdv2.intel.com/v1/dl/getContent/671368
Or I am missing something?
Yes, it is not clearly described in LAM spec.
Had some internal discussions and also did some tests in host, if the processor supports LAM, CR4.LAM_SUP is allowed to be set even when cpu isn't in 64bit mode.Quoted 4 lines
There was a statement in commit message of the last version, but I missed it in this version. I'll add it back.
"CR4.LAM_SUP is allowed to be set even not in 64-bit mode, but it will not take effect since LAM only applies to 64-bit linear address."
Quoted 5 lines
Yeah this does help. Please add it back to the changelog.
Quoted 2 lines
Also, I will try to ask Intel guys if it's possible to update the document.
Thanks.
Per the internal discussion, there is no need to explicitly callout CR4[28] can be set out side of 64-bit mode in SDM/LAM spec for the following reasons:
According to SDM Vol.2 Move to/from Control Registers:
- "On a 64-bit capable processor, an execution of MOV to CR outside of 64-bit mode zeros the upper 32 bits of the control register." It doesn't mention of clearing any of the lower bits.
- "Some of the bits in CR0, CR3, and CR4 are reserved and must be written with zeros. ... Attempting to set any reserved bits in CR4 results in #GP(0)" CR4[28] is not reserved on processors that support LAM, and SDM / LAM spec doesn't explicitly say the bit cannot be set under some specific condition.
So just like the reset of CR4[31:0], CR4[28] can be set by any 32-bit load of CR4 when LAM is supported.
For example, CR4[17] is used only with 64-bit paging, but it can be set by a 32-bit load of CR4 when 32-bit paging or PAE paging is in use. Similarly, user-interrupt delivery is enabled by setting CR4[25]. It can be set in any mode, even though user-interrupt delivery can occur only in 64-bit mode.
Gentle ping.
Do you have any comments for this patch series?
Thanks.
Quoted 205 lines · Binbin Wu
===Feature Introduction===
Linear-address masking (LAM) [1], modifies the checking that is applied to *64-bit* linear addresses, allowing software to use of the untranslated address bits for metadata and masks the metadata bits before using them as linear addresses to access memory.
When the feature is virtualized and exposed to guest, it can be used for efficient address sanitizers (ASAN) implementation and for optimizations in JITs and virtual machines.
Regarding which pointer bits are masked and can be used for metadata, LAM has 2 modes:
-LAM_48: metadata bits 62:48, i.e. LAM width of 15.
-LAM_57: metadata bits 62:57, i.e. LAM width of 6.* For user pointers:
CR3.LAM_U57 = CR3.LAM_U48 = 0, LAM is off; CR3.LAM_U57 = 1, LAM57 is active; CR3.LAM_U57 = 0 and CR3.LAM_U48 = 1, LAM48 is active. * For supervisor pointers: CR4.LAM_SUP =0, LAM is off; CR4.LAM_SUP =1 with 5-level paging mode, LAM57 is active; CR4.LAM_SUP =1 with 4-level paging mode, LAM48 is active.The modified LAM canonicality check:
*LAM_S48: [ 1 ][ metadata ][ 1 ]
63 47
*LAM_U48: [ 0 ][ metadata ][ 0 ]
63 47
*LAM_S57: [ 1 ][ metadata ][ 1 ]
63 56
*LAM_U57+ 5-lvl paging : [ 0 ][ metadata ][ 0 ]
63 56
*LAM_U57+ 4-lvl paging : [ 0 ][ metadata ][ 0...0 ]63 56..47Note:
1. LAM applies to only data address, not to instructions.
2. LAM identification of an address as user or supervisor is based solely on the value of pointer bit 63 and does not depend on the CPL.
3. LAM doesn't apply to the writes to control registers or MSRs.
4. LAM masking applies before paging, so the faulting linear address in CR2 doesn't contain the metadata.
5 The guest linear address saved in VMCS doesn't contain metadata.
6. For user mode address, it is possible that 5-level paging andLAM_U48are both set, in this case, the effective usable linear address width is 48. (Currently, onlyLAM_U57is enabled in Linux kernel. [2])===LAM KVM Design===
LAM KVM enabling includes the following parts:
- Feature Enumeration
LAM feature is enumerated by CPUID.7.1:EAX.LAM[bit 26].
If hardware supports LAM and host doesn't disable it explicitly (e.g. via clearcpuid), LAM feature will be exposed to user VMM.- CR4 Virtualization
LAM uses CR4.LAM_SUP (bit 28) to configure LAM on supervisor pointers. Add support to allow guests to set the new CR4 control bit for guests to enable LAM on supervisor pointers.- CR3 Virtualization
LAM uses CR3.LAM_U48 (bit 62) and CR3.LAM_U57 (bit 61) to configure LAM on user pointers.
Add support to allow guests to set two new CR3 non-address control bits for guests to enable LAM on user pointers.- Modified Canonicality Check and Metadata Mask
When LAM is enabled, 64-bit linear address may be tagged with metadata. Linear address should be checked for modified canonicality and untagged in instruction emulations and VMExit handlers when LAM is applicable.LAM inside nested guest is supported by this patch series. LAM inside SGX enclave mode is NOT supported by this patch series.
LAM QEMU patch:
https://lists.gnu.org/archive/html/qemu-devel/2023-02/msg08036.htmlLAM kvm-unit-tests patch:
https://lore.kernel.org/kvm/[email protected]/===Test===
1. Add test cases in kvm-unit-test [3] for LAM, includingLAM_SUPand LAM_{U57,U48}. For supervisor pointers, the test covers CR4LAM_SUPbits toggle, Memory/MMIO access with tagged pointer, and some special instructions (INVLPG, INVPCID, INVVPID), INVVPID cases also used to cover VMX instruction VMExit path. For uer pointers, the test covers CR3 LAM bits toggle, Memory/MMIO access with tagged pointer.
MMIO cases are used to trigger instruction emulation path.
Run the unit test with both LAM feature on/off (i.e. including negative cases). Run the unit test in L1 guest with both LAM feature on/off.
2. Run Kernel LAM kselftests [2] in guest, with both EPT=Y/N.
3. Launch a nested guest.All tests have passed in Simics environment.
[1] Intel ISE https://cdrdv2.intel.com/v1/dl/getContent/671368
Chapter Linear Address Masking (LAM)
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/kvm/[email protected]/---
Changelog
v7 --> v8:
- Rebase the patch series on Linux kernel v6.4-rc1, which has all dependencies of this patch series.
- Add a cleanup patch to consolidate flags for__linearize().
- Some changes in commit message of patch 2.
- Rewrite the commit message of patch 3 (Thanks Kai Huang for the suggestion). Opportunistically useGENMASK_ULL()to define__PT_BASE_ADDR_MASK. Opportunistically usekvm_vcpu_is_legal_cr3()to check CR3 in SVM nested code, to provide a clear distinction b/t CR3 and GPA checks.
- Change interfaceuntag_addr()to optional version to avoid adding a dummy version in SVM in patch 4.
- Verified and confirmed that LAM doesn't apply to the address in descriptor of invvpid, drop the untag action for it in patch 5.v6 --> v7:
- Changes to CR3 virtualization when EPT off
* Leave LAM bits in root.pgd to force a new root for a CR3+LAM combination. (Sean)
* Perform GFN calculation from guest CR3/PGD generically by extracting the maximal base address mask. (Sean)
- Remove de-reference ofctxt->vcpuin the emulator. (Sean)
- Fix a bug in v6, which hardcoded "write" to "false" by mistake inlinearize(). (Chao Gao)
- Add Chao Gao's reviewed-by in Patch 5.
- Add Xuelian Guo's tested-by in the patch set.
- Separate cleanup patches from the patch set.v5 --> v6:
Add Patch 2 to fix the check of 64-bit mode.
Adduntag_addr()tokvm_x86_opsto hide vendor specific code.
Simplify the LAM canonicality check per Chao's suggestion.
Addcr3_ctrl_bitstokvm_vcpu_archto simplify cr3 invalidation/extract/mask (Chao Gao) Extend the patchset scope to include nested virtualization and SGX ENCLS handling.
- AddX86_CR4_LAM_SUPincr4_fixed1_updatefor nested vmx. (Chao Gao)
- Add SGX ENCLS VMExit handling
- Add VMX instruction VMExit handling
More descriptions in cover letter.
Add Chao's reviewed-by on Patch 4.
Add more test cases in kvm-unit-test.v4 --> v5:
Reorder and melt patches surround CR3.LAM bits into Patch 3 of this version.
Revise Patch 1's subject and description
Drop Patch 3
Usekvm_read_cr4_bits()instead ofkvm_read_cr4()
Fix: No need to untag addr when write to msr, it should be legacy canonical check Renamekvm_is_valid_cr3()-->kvm_vcpu_is_valid_cr3(), and update some call sites ofkvm_vcpu_is_valid_cr3()to usekvm_is_valid_cr3().
Other refactors and Miscs.v3 --> v4:
Drop unrelated Patch 1 in v3 (Binbin, Sean, Xiaoyao)
Intercept CR4.LAM_SUP instead of pass through to guest (Sean) Just filter out CR3.LAM_{U48, U57}, instead of all reserved high bits (Sean, Yuan)
Use existing__canonical_address()helper instead write a new one (Weijiang) Add LAM handling in KVM emulation (Yu, Yuan)
Add Jingqi's reviwed-by on Patch 7
Rebased to Kirill's latest code, which is 6.2-rc1 base.v2 --> v3:
As LAM Kernel patches are in tip tree now, rebase to it.
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/v1 --> v2:
1. Fixes i386-allyesconfig build error onget_pgd(), whereCR3_HIGH_RSVD_MASKisn't applicable.
(Reported-by: kernel test robot <[email protected]>)
2. Inkvm_set_cr3(), be conservative on skip tlb flush when only LAM bits toggles. (Kirill)Binbin Wu (3):
KVM: x86: Consolidate flags for__linearize()
KVM: x86: Introduceuntag_addr()inkvm_x86_ops
KVM: x86: Untag address when LAM applicableRobert Hoo (3):
KVM: x86: Virtualize CR4.LAM_SUP
KVM: x86: Virtualize CR3.LAM_{U48,U57}
KVM: x86: Expose LAM feature to userspace VMMarch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 10 ++++- arch/x86/kvm/cpuid.c | 2 +- arch/x86/kvm/cpuid.h | 5 +++ arch/x86/kvm/emulate.c | 30 +++++++++++---- arch/x86/kvm/kvm_emulate.h | 7 ++++ arch/x86/kvm/mmu.h | 5 +++ arch/x86/kvm/mmu/mmu.c | 8 +++- arch/x86/kvm/mmu/mmu_internal.h | 1 + arch/x86/kvm/mmu/paging_tmpl.h | 3 +- arch/x86/kvm/mmu/spte.h | 2 +- arch/x86/kvm/svm/nested.c | 4 +- arch/x86/kvm/vmx/nested.c | 6 ++- arch/x86/kvm/vmx/sgx.c | 1 + arch/x86/kvm/vmx/vmx.c | 59 +++++++++++++++++++++++++++++- arch/x86/kvm/vmx/vmx.h | 2 + arch/x86/kvm/x86.c | 11 +++++- arch/x86/kvm/x86.h | 2 + 18 files changed, 140 insertions(+), 19 deletions(-)base-commit:
ac9a78681b921877518763ba0e89202254349d1b
Sean Christopherson
maintainerQuoted 2 lines · Binbin Wu
Gentle ping.
Do you have any comments for this patch series?
Probably, I haven't looked at it yet. This is on my todo list, but there are multiple large series ahead of it.
Quoted 5 lines · Sean Christopherson
Quoted 2 lines · Binbin Wu
Gentle ping.
Do you have any comments for this patch series?Probably, I haven't looked at it yet. This is on my todo list, but there are multiple large series ahead of it.
Updated the patch version to v9 to address the comments from Chao, Kai and David in v8.
https://lore.kernel.org/kvm/[email protected]/T/#t