KVM: TDX: Validate directly configurable CPUID bits
Subsystems (4)
v3 cover KVM: TDX: Validate directly configurable CPUID bits
Hi,
The purpose of this patch series is to prevent userspace from enabling host state clobbering features that KVM does not support for TDX. A host state clobbering feature exposed on a new TDX module/platform can corrupt host state if KVM does not explicitly save and restore the related MSR(s) across host/guest transitions. If such a feature is blindly exposed to and used by a TD, the host will behave unexpectedly.
Except for a few fixed-1 bits required for basic TDX support, host state clobbering features are either directly configurable or gated by TD ATTRIBUTES/XFAM. So an allowlist covering only the directly configurable CPUID bits, plus the corresponding filtering and validation, is sufficient to serve the purpose while keeping the code footprint small.
Open question for Sean/Paolo
============================
This revision comes long after Sean confirmed the overall direction on RFC v2 [1], due to internal discussions about a potential backward compatibility issue: a CPUID field can change virtualization type to directly configurable. The TDX module can maintain backward compatibility in most cases; the problematic case is a single bit that was effectively "1" and non-configurable becoming directly configurable. The only scenario we can envision for that is an x86 feature deprecation.
In that case the TDX module has to decide how to handle deprecation, especially with respect to TD migration compatibility. That pre-existing VMs with the deprecated feature enabled cannot migrate from an old platform to a new one is a problem shared by TDX and VMX, but the TDX module additionally has to decide:
- Whether to allow TDs newly created on old platforms to migrate to new platforms.
- Whether to allow TDs created on new platforms to migrate to old platforms.
There are two options:
1. Do not offer the VMM the ability to configure the bit, i.e. simply report it as zero on platforms that no longer support the feature. This disallows both migration cases above.
2. Make the bit directly configurable so the VMM can clear it even on platforms that still support the feature, allowing both migration cases above. However, since the bit is not in KVM's allowlist, userspace would no longer be able to enable it on old platforms after a TDX module update.
Adding all supported CPUID bits to the allowlist, regardless of their virtualization type in the TDX module, would avoid the compatibility issue for option 2, at the cost of a significantly larger series.
Given that x86 feature deprecation is expected to be very rare, and that Intel will discuss such cases with the community beforehand, does the approach taken by this series look acceptable? If so, the goal for this revision is to collect detailed feedback and, where appropriate, Reviewed-by tags.
Expected host state clobbering behavior for TDX
===============================================
We also want to call for discussions about the expected host state clobbering behavior for TDX here for future features.
For a normal VMX guest, VM entry/exit behavior for a given piece of CPU state is architecturally defined: state is either switched by hardware via VMCS host/guest fields, or left as the guest value on VM exit and managed by KVM in software.
For TDs, the host/guest transition goes through TDH.VP.ENTER, and what the TDX module does with a given piece of host state is defined by the TDX module ABI rather than by the x86 architecture.
What we would like to align on is the expected baseline behavior of TDH.VP.ENTER for future features. The proposal is to have TDX simply match VMX behavior, i.e. on return from TDH.VP.ENTER, state that VMX would restore from the VMCS host fields is restored, and state that VMX would leave as the guest value is clobbered. That keeps a single model for VMM, and means enabling a new feature for TDs requires the same work flow as enabling it for VMX.
FRED is a useful concrete example. Under VMX, the FRED host state in IA32_FRED_CONFIG, IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX module is expected to restore these MSRs on TDH.VP.ENTER return. IA32_FRED_RSP0 and IA32_PL0_SSP (a.k.a. IA32_FRED_SSP0) are handled by software, so the TDX module is expected to clobber them on TDH.VP.ENTER return.
The series
==========
Starting with v2 [2], the series takes a simpler, TDX-contained approach instead of the comprehensive CPUID paranoid verification framework across VMX, SVM, and TDX proposed in v1 [3]. It validates only the TDX directly configurable CPUID bits, which are reported by the TDX module in CPUID_CONFIG fields that the VMM can configure for a TD. All filtering and validation logic is isolated within TDX code. This is sufficient to address the host clobbering issue because:
- The TDX module will not introduce new host state clobbering features that are fixed-1.
- Non-directly configurable feature bits (i.e. features controlled by XFAM or ATTRIBUTES) cannot be enabled by userspace without KVM's support.
Specifically, this series builds a KVM-side allowlist of supported TDX directly configurable CPUID bits to:
- Filter KVM_TDX_CAPABILITIES
Replace the hardcoded denylist to only report configurable bits that KVM explicitly supports.
- Validate KVM_TDX_INIT_VM
Reject any configurable bit that the TDX module allows but KVM does not support, as well as CPUID entries with an unexpected subleaf.
With this allowlist, newly added TDX configurable CPUID bits will not be exposed to userspace until KVM explicitly opts in after fulfilling the necessary virtualization requirements.
The allowlist consists of two parts:
- Feature CPUID bits, which are tracked in tdx_cpu_cfg_caps[] following the organization of kvm_cpu_caps[]. It holds KVM's supported masks for the TDX configurable CPUID feature bits.
- Non-feature multi-bit fields, which are handled at runtime for the CPUID registers holding such fields.
For compatibility with older TDX modules that report CORE_CAPABILITIES as fixed-1, report CORE_CAPABILITIES as configurable even though KVM does not support guest access to MSR_IA32_CORE_CAPS. This allows userspace to keep enabling CORE_CAPABILITIES when the bit changes from fixed-1 to directly configurable, and allows userspace to infer that the bit is no longer fixed-1 so it can adjust its expectations.
Known limitation:
- The series does not check KVM_SET_CPUID2 input for consistency with the CPUID configuration supplied through KVM_TDX_INIT_VM. This is not required for host safety today because KVM does not use its vCPU CPUID model to decide whether to manage host-clobbering state for TDs. A consistency check should be added if KVM ever starts relying on the vCPU CPUID model for such decisions.
Changes from v2 [2]:
- Drop the dedicated data structure introduced in v2; track only feature bits following the organization of kvm_cpu_caps[], and handle non-feature bits at runtime. (Sean)
- Add AMX_COMPLEX since it has been defined in the CPUID virtualization doc.
- Add CPUID.0x24.0.EBX[7:0] into allow list. There is a mismatch of the description about CPUID.0x24.0.EBX[7:0], which is listed as "XFAM & CPUID_Enabled & Native" but should be "XFAM & CPUID_Enabled & Configured & Native".
- Move the patch for CORE_CAPABILITIES earlier in the series to avoid breaking userspace during bisection. (Xiaoyao)
- Report CORE_CAPABILITIES as a configurable bit to userspace for backward compatibility, but drop the MSR_IA32_CORE_CAPS access code.
- Use two versions of macros to distinguish whether a supported TDX configurable CPUID bit should be checked against KVM's common CPU capabilities.
- Validate KVM_TDX_INIT_VM input against the allowlist instead of masking it, and additionally reject entries whose index does not match the TDX sysinfo configuration. (Sashiko)
Changes from v1 [3]:
- Drop the overarching CPUID paranoid verification framework across VMX/SVM/TDX and the opt-in interface. (Sean)
- Shift focus entirely to isolating and validating TDX directly configurable CPUID bits.
The series follows the CSV version of the CPUID virtualization document from "Intel TDX Module ABI Definitions" [4], updated in June 2026.
AI tools were used to:
- Check the coverage of the directly configurable CPUID bits in the document against the allowlist in this series.
- Polish the cover letter and commit messages.
- Do code review.
[1] https://lore.kernel.org/kvm/[email protected]
[2] https://lore.kernel.org/kvm/[email protected]
[3] https://lore.kernel.org/kvm/[email protected]
[4] https://cdrdv2.intel.com/v1/dl/getContent/795381
Binbin Wu (4):
KVM: TDX: Track configurable CPUID bits allowed by KVM
KVM: TDX: Report CORE_CAPABILITIES as configurable
KVM: TDX: Filter configurable CPUID bits
KVM: TDX: Validate userspace CPUID input for KVM_TDX_INIT_VM
arch/x86/kvm/vmx/tdx.c | 258 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 229 insertions(+), 29 deletions(-)
base-commit: 76671054f9a1ff6abb976583cd8da37650acdc97
Signature · 1 line
2.46.0
PATCH v3 1/4 KVM: TDX: Track configurable CPUID bits allowed by KVM
Add tdx_cpu_cfg_caps[] to track the subset of TDX directly configurable CPUID feature bits that KVM supports, and build the masks during TDX hardware setup via tdx_initialize_cpu_cfg_caps().
The TDX module reports the CPUID bits that the VMM can directly configure for a TD, but KVM cannot blindly expose all reported bits to userspace. Certain features imply additional architectural state, e.g. one or more MSRs, that KVM must explicitly manage across host/guest transitions to prevent host state corruption.
Today KVM relies on a hardcoded denylist, i.e. it clears a few known problematic bits, e.g. TSX and WAITPKG, and passes everything else through. A denylist is fundamentally fragile while an allowlist inverts the default, i.e. unknown configurable bits are hidden and not allowed to be enabled until KVM explicitly opts in.
Except for a few fixed-1 bits required for basic TDX support, host state clobbering features are either directly configurable or gated by TD ATTRIBUTES/XFAM. Tracking only the directly configurable feature bits is therefore sufficient to serve the purpose while keeping the code footprint small.
Organize tdx_cpu_cfg_caps[] following kvm_cpu_caps[] so that the masks can be built with the similar feature-name based initializers. CPUID registers that hold directly configurable non-feature (multi-bit) fields are handled separately.
The allowlist is consumed by later patches to filter KVM_TDX_CAPABILITIES and to reject unsupported CPUID input to KVM_TDX_INIT_VM, so that newly introduced TDX directly configurable CPUID feature bits stay hidden from userspace until KVM explicitly opts in.
Add comments as placeholders for HLE, RTM and WAITPKG, which KVM doesn't support for TDX yet.
---
v3:
- Drop the new data structure in v2 and only track feature bits by following the organization of kvm_cpu_caps[], handle non-feature bits separately. (Sean)
- Use two versions of macros (TDX_CFG_F() VS. TDX_CFG_EXTRA_F()) to distinguish whether a supported TDX configurable CPUID bit should be checked against KVM's common cpu capabilities.
- Add AMX_COMPLEX since it has been defined in the CPUID virtualization doc.
--- arch/x86/kvm/vmx/tdx.c | 145 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+)
arch/x86/kvm/vmx/tdx.c +145 -0
@@ -52,6 +52,149 @@ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #a3 " 0x%llx", \ a1, a2, a3) +static u32 tdx_cpu_cfg_caps[NR_KVM_CPU_CAPS ]__ro_after_init ; +static_assert (ARRAY_SIZE (tdx_cpu_cfg_caps) ==ARRAY_SIZE (kvm_cpu_caps)); + +#define TDX_VALIDATE_CPU_CAP_USAGE (name) \ + BUILD_BUG_ON(__feature_leaf(X86_FEATURE_##name) != \ + tdx_cpu_cap_init_in_progress) + +/* For feature bit that KVM advertised through kvm_cpu_caps[]. */ +#define TDX_CFG_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_caps |= feature_bit(name); \ +}) + +/* +* For feature bit KVM allows for TDX guests even though it is not advertised +* through kvm_cpu_caps[], e.g. MWAIT. +*/ +#define TDX_CFG_EXTRA_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_extra_caps |= feature_bit(name); \ +}) + +#define tdx_cpu_cfg_cap_init(leaf, feature_initializers...) \ +do { \ + const u32 __maybe_unused tdx_cpu_cap_init_in_progress = leaf; \ + u32 tdx_cfg_extra_caps = 0; \ + u32 tdx_cfg_caps = 0; \ + \ + feature_initializers \ + tdx_cpu_cfg_caps[leaf] = (tdx_cfg_caps & kvm_cpu_caps[leaf]) | \ + tdx_cfg_extra_caps; \ +} while (0) + +/* +* Track only CPUID feature bits that are directly configurable by userspace. +* Features controlled by XFAM or ATTRIBUTES are excluded; userspace cannot +* enable them until KVM adds support for the corresponding control. +*/ +static void __init tdx_initialize_cpu_cfg_caps (void ) +{ +tdx_cpu_cfg_cap_init (CPUID_1_ECX , +TDX_CFG_EXTRA_F (MWAIT ), +TDX_CFG_F (TSC_DEADLINE_TIMER ), +TDX_CFG_F (AVX ), + TDX_CFG_F(F16C), + ); + +tdx_cpu_cfg_cap_init (CPUID_1_EDX , +TDX_CFG_F (MCE ), +TDX_CFG_F (MTRR ), +TDX_CFG_F (MCA ), + TDX_CFG_F(SELFSNOOP), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_0_EBX , +TDX_CFG_F (BMI1 ), +/* HLE */ +TDX_CFG_F (BMI2 ), +TDX_CFG_F (ERMS ), +/* RTM */ +TDX_CFG_F (AVX512F ), +TDX_CFG_F (AVX512DQ ), +TDX_CFG_F (ADX ), +TDX_CFG_F (AVX512IFMA ), +TDX_CFG_F (AVX512PF ), +TDX_CFG_F (AVX512ER ), +TDX_CFG_F (AVX512CD ), +TDX_CFG_F (AVX512BW ), + TDX_CFG_F(AVX512VL), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_ECX , +TDX_CFG_F (UMIP ), +/* WAITPKG */ +TDX_CFG_F (AVX512_VBMI2 ), +TDX_CFG_F (GFNI ), +TDX_CFG_F (VAES ), +TDX_CFG_F (VPCLMULQDQ ), +TDX_CFG_F (AVX512_VNNI ), +TDX_CFG_F (AVX512_BITALG ), +TDX_CFG_F (AVX512_VPOPCNTDQ ), +TDX_CFG_F (LA57 ), +TDX_CFG_F (RDPID ), + TDX_CFG_F(CLDEMOTE), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_EDX , +TDX_CFG_F (AVX512_4VNNIW ), +TDX_CFG_F (AVX512_4FMAPS ), +TDX_CFG_F (FSRM ), +TDX_CFG_F (AVX512_VP2INTERSECT ), +TDX_CFG_F (SERIALIZE ), + TDX_CFG_F(TSXLDTRK), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_1_EAX , +TDX_CFG_F (SHA512 ), +TDX_CFG_F (SM3 ), +TDX_CFG_F (SM4 ), +TDX_CFG_F (AVX_VNNI ), +TDX_CFG_F (AVX512_BF16 ), +TDX_CFG_F (CMPCCXADD ), +TDX_CFG_F (FZRM ), +TDX_CFG_F (FSRS ), +TDX_CFG_F (FSRC ), +TDX_CFG_F (LKGS ), +TDX_CFG_F (WRMSRNS ), +TDX_CFG_F (AMX_FP16 ), +TDX_CFG_F (AVX_IFMA ), +TDX_CFG_F (LAM ), + TDX_CFG_F(MOVRS), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_1_EDX , +TDX_CFG_F (AVX_VNNI_INT8 ), +TDX_CFG_F (AVX_NE_CONVERT ), +TDX_CFG_F (AMX_COMPLEX ), +TDX_CFG_F (AVX_VNNI_INT16 ), +TDX_CFG_F (PREFETCHITI ), + TDX_CFG_F(AVX10), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_2_EDX , +TDX_CFG_F (DDPD_U ), + TDX_CFG_F(MCDT_NO), + ); + +tdx_cpu_cfg_cap_init (CPUID_1E_1_EAX , +TDX_CFG_F (AMX_FP8 ), +TDX_CFG_F (AMX_TF32 ), +TDX_CFG_F (AMX_AVX512 ), + TDX_CFG_F(AMX_MOVRS), + ); + +tdx_cpu_cfg_cap_init (CPUID_8000_0008_EBX , + TDX_CFG_F(WBNOINVD), + ); +} + +#undef TDX_CFG_F +#undef TDX_CFG_EXTRA_Fbool enable_tdx__ro_after_init ;module_param_named (tdx, enable_tdx, bool,0444 );
@@ -3481,6 +3624,8 @@ int __init tdx_hardware_setup(void)return r; } +tdx_initialize_cpu_cfg_caps (); +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_tdx); vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size, sizeof(struct kvm_tdx));
Signature · 1 line
2.46.0
PATCH v3 2/4 KVM: TDX: Report CORE_CAPABILITIES as configurable
In reply to Binbin WuAdd CORE_CAPABILITIES (CPUID.0x7.0.EDX[30]) to KVM's allowlist of TDX directly configurable CPUID feature bits, even though KVM doesn't support MSR_IA32_CORE_CAPS for TDX guests, to accommodate legacy TDX module behavior.
Older TDX specs define the CORE_CAPABILITIES CPUID bit as fixed-1, so userspace may expect the bit to be enabled for TDs. If the bit becomes directly configurable in a newer TDX module but is not reported as such to userspace, userspace can no longer enable it once KVM starts validating the CPUID configuration input.
Reporting CORE_CAPABILITIES as configurable keeps userspace able to enable the bit across the fixed-1 => configurable transition, and lets userspace infer that the bit is no longer fixed-1 so it can adjust its expectations.
Keep MSR_IA32_CORE_CAPS unsupported for TDX guests, as existing TDX users have not needed guest access to the MSR, and advertising the CPUID bit as configurable is enough for userspace to handle the legacy-module compatibility case.
---
v3:
- Move this patch earlier in the series to avoid breaking userspace during bisection. (Xiaoyao)
- Drop the code for MSR_IA32_CORE_CAPS access.
--- arch/x86/kvm/vmx/tdx.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -147,6 +147,12 @@ static void __init tdx_initialize_cpu_cfg_caps(void)TDX_CFG_F (AVX512_VP2INTERSECT ),TDX_CFG_F (SERIALIZE ),TDX_CFG_F (TSXLDTRK ), +/* +* KVM does not support MSR_IA32_CORE_CAPS, but older TDX specs +* define this bit as fixed-1. Report it as configurable so +* userspace can know the feature is no longer a fixed-1 bit. +*/ +TDX_CFG_EXTRA_F (CORE_CAPABILITIES ), ); tdx_cpu_cfg_cap_init(CPUID_7_1_EAX,
Signature · 1 line
2.46.0
PATCH v3 3/4 KVM: TDX: Filter configurable CPUID bits
In reply to Binbin WuFilter the directly configurable CPUID bits reported through KVM_TDX_CAPABILITIES against KVM's TDX allowlist, and drop the hardcoded denylist based filtering.
The TDX module reports all directly configurable CPUID bits that it supports for a TD, but KVM must not expose bits that it doesn't support, as blindly exposing a host state clobbering feature can lead to host state corruption. The existing denylist, which clears only TSX and WAITPKG, is not fail-safe.
Add tdx_get_allowed_cfg_cpuid_mask() to get the mask of directly configurable bits allowed by KVM for a given CPUID register, covering both feature bits, which come from tdx_cpu_cfg_caps[], and non-feature bits, which are enumerated at runtime. Apply the mask to every CPUID register reported through KVM_TDX_CAPABILITIES.
With the allowlist in place, newly introduced TDX directly configurable CPUID bits stay hidden from userspace until KVM explicitly opts in.
---
v3:
- Handle non-feature leafs at runtime. (Sean)
- Add CPUID.0x24.0.EBX[7:0] into allow list. There is a mismatch of the description about CPUID.0x24.0.EBX[7:0], which is listed as "XFAM & CPUID_Enabled & Native" but should be "XFAM & CPUID_Enabled & Configured & Native".
--- arch/x86/kvm/vmx/tdx.c | 84 +++++++++++++++++++++++++++++++++--------- 1 file changed, 66 insertions(+), 18 deletions(-)
arch/x86/kvm/vmx/tdx.c +66 -18
@@ -277,34 +277,76 @@ static bool has_tsx(const struct kvm_cpuid_entry2 *entry) (entry->ebx &TDX_FEATURE_TSX ); } -static void clear_tsx (struct kvm_cpuid_entry2 *entry) -{ - entry->ebx &= ~TDX_FEATURE_TSX ; -} -static bool has_waitpkg (const struct kvm_cpuid_entry2 *entry) {return entry->function ==7 && entry->index ==0 && (entry->ecx &__feature_bit (X86_FEATURE_WAITPKG )); } -static void clear_waitpkg (struct kvm_cpuid_entry2 *entry) +static bool tdx_unsupported_cpuid (const struct kvm_cpuid_entry2 *entry) { - entry->ecx &= ~__feature_bit (X86_FEATURE_WAITPKG ); +return has_tsx (entry) ||has_waitpkg (entry); } -static void tdx_clear_unsupported_cpuid (struct kvm_cpuid_entry2 *entry) +#define TDX_CPUID_ALL_ALLOWED_MASK GENMASK_U32(31, 0) + +static u32 tdx_cfg_non_feature_mask (u32 function,u32 index,int reg) { -if (has_tsx (entry)) -clear_tsx (entry); +/* +* For a leaf/subleaf/register that will never be repurposed to hold +* feature bits, it's safe to return TDX_CPUID_ALL_ALLOWED_MASK, i.e. +* leave the TDX module's CPUID config mask intact. +*/ +switch (function) { +case 1 : +if (reg ==CPUID_EAX || reg ==CPUID_EBX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +case 4 : +case 0x18 : +case 0x1f : +return TDX_CPUID_ALL_ALLOWED_MASK ; +case 0x24 : +if (index ==0 && reg ==CPUID_EBX ) +return GENMASK_U32 (7 ,0 ); +return 0 ; +case 0x80000008 : +if (reg ==CPUID_EAX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +default : +return 0 ; + } +} -if (has_waitpkg (entry)) -clear_waitpkg (entry); +static u32 tdx_cfg_feature_mask (u32 function,u32 index,int reg) +{ +for (int i =0 ; i <NR_KVM_CPU_CAPS ; i++) { +const struct cpuid_reg *cpuid = &reverse_cpuid[i]; + +if (!cpuid->function) +continue ; + +if (cpuid->function == function && cpuid->index == index && + cpuid->reg == reg) +return tdx_cpu_cfg_caps[i]; + } + +return 0 ; } -static bool tdx_unsupported_cpuid (const struct kvm_cpuid_entry2 *entry) +static u32 tdx_get_allowed_cfg_cpuid_mask (u32 function,u32 index,int reg) { -return has_tsx (entry) ||has_waitpkg (entry); +u32 non_feature_mask =tdx_cfg_non_feature_mask (function, index, reg); + +if (non_feature_mask ==TDX_CPUID_ALL_ALLOWED_MASK ) +return TDX_CPUID_ALL_ALLOWED_MASK ; + +/* +* It's possible that a CPUID register contains both feature and +* non-feature bits. +*/ +return non_feature_mask |tdx_cfg_feature_mask (function, index, reg); }#define KVM_TDX_CPUID_NO_SUBLEAF ((__u32)-1)
@@ -330,8 +372,6 @@ static void td_init_cpuid_entry2(struct kvm_cpuid_entry2 *entry, unsigned char i*/ if (entry->function ==0x80000008 ) entry->eax =tdx_set_guest_phys_addr_bits (entry->eax,0xff ); - -tdx_clear_unsupported_cpuid (entry); }#define TDVMCALLINFO_SETUP_EVENT_NOTIFY_INTERRUPT BIT(1)
@@ -354,8 +394,16 @@ static int init_kvm_tdx_caps(const struct tdx_sys_info_td_conf *td_conf, caps->user_tdvmcallinfo_1_r11 =TDVMCALLINFO_SETUP_EVENT_NOTIFY_INTERRUPT ; -for (i =0 ; i < td_conf->num_cpuid_config; i++) -td_init_cpuid_entry2 (&caps->cpuid.entries[i], i); +for (i =0 ; i < td_conf->num_cpuid_config; i++) { +struct kvm_cpuid_entry2 *e = &caps->cpuid.entries[i]; + +td_init_cpuid_entry2 (e, i); +/* Only report the configurable bits allowed by KVM. */ + e->eax &=tdx_get_allowed_cfg_cpuid_mask (e->function, e->index,CPUID_EAX ); + e->ebx &=tdx_get_allowed_cfg_cpuid_mask (e->function, e->index,CPUID_EBX ); + e->ecx &=tdx_get_allowed_cfg_cpuid_mask (e->function, e->index,CPUID_ECX ); + e->edx &=tdx_get_allowed_cfg_cpuid_mask (e->function, e->index,CPUID_EDX ); + }return 0 ; }
Signature · 1 line
2.46.0
PATCH v3 4/4 KVM: TDX: Validate userspace CPUID input for KVM_TDX_INIT_VM
In reply to Binbin WuValidate the CPUID configuration provided by userspace through KVM_TDX_INIT_VM against KVM's TDX allowlist, and drop the hardcoded denylist based check.
The TDX module lets the VMM configure certain CPUID features for a TD at initialization time, but KVM must strictly govern which of them userspace can actually enable, otherwise a host state clobbering feature could be enabled behind KVM's back. The existing check only rejects TSX and WAITPKG, i.e. it is not fail-safe, as any bit that a future TDX module makes configurable would be accepted even if KVM has no idea about the feature.
Add tdx_has_unsupported_cfg_cpuid_bit() and reject KVM_TDX_INIT_VM if userspace sets any bit outside the mask returned by tdx_get_allowed_cfg_cpuid_mask(). There is no need to first mask the userspace input with the bits the TDX module reports as directly configurable, as anything outside that set is rejected by the TDX module itself.
Also reject CPUID entries whose index differs from the value expected by the TDX module, as kvm_find_cpuid_entry2() ignores the index when KVM_CPUID_FLAG_SIGNIFCANT_INDEX is cleared, and so a mismatching entry could otherwise be applied to the wrong subleaf.
---
v3:
- Check CPUID entry index mismatch b/t userspace input and TDX sysinfo configuration. (Sashiko)
- No need to mask the userspace input with TDX module reported directly configurable bits first, since the userspace input should be subset of directly configurable bits. Otherwise, the input will be rejected by the TDX module.
--- arch/x86/kvm/vmx/tdx.c | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-)
arch/x86/kvm/vmx/tdx.c +21 -20
@@ -269,25 +269,6 @@ static u32 tdx_set_guest_phys_addr_bits(const u32 eax, int addr_bits)return (eax & ~GENMASK (23 ,16 )) | (addr_bits &0xff ) <<16 ; } -#define TDX_FEATURE_TSX (__feature_bit(X86_FEATURE_HLE) | __feature_bit(X86_FEATURE_RTM)) - -static bool has_tsx (const struct kvm_cpuid_entry2 *entry) -{ -return entry->function ==7 && entry->index ==0 && - (entry->ebx &TDX_FEATURE_TSX ); -} - -static bool has_waitpkg (const struct kvm_cpuid_entry2 *entry) -{ -return entry->function ==7 && entry->index ==0 && - (entry->ecx &__feature_bit (X86_FEATURE_WAITPKG )); -} - -static bool tdx_unsupported_cpuid (const struct kvm_cpuid_entry2 *entry) -{ -return has_tsx (entry) ||has_waitpkg (entry); -} -#define TDX_CPUID_ALL_ALLOWED_MASK GENMASK_U32(31, 0)static u32 tdx_cfg_non_feature_mask (u32 function,u32 index,int reg)
@@ -2533,6 +2514,17 @@ static int setup_tdparams_eptp_controls(struct kvm_cpuid2 *cpuid,return 0 ; } +static bool tdx_has_unsupported_cfg_cpuid_bit (const struct kvm_cpuid_entry2 *entry) +{ +u32 function = entry->function; +u32 index = entry->index; + +return (entry->eax & ~tdx_get_allowed_cfg_cpuid_mask (function, index,CPUID_EAX )) || + (entry->ebx & ~tdx_get_allowed_cfg_cpuid_mask (function, index,CPUID_EBX )) || + (entry->ecx & ~tdx_get_allowed_cfg_cpuid_mask (function, index,CPUID_ECX )) || + (entry->edx & ~tdx_get_allowed_cfg_cpuid_mask (function, index,CPUID_EDX )); +} +static int setup_tdparams_cpuids (struct kvm_cpuid2 *cpuid,struct td_params *td_params) {
@@ -2556,7 +2548,16 @@ static int setup_tdparams_cpuids(struct kvm_cpuid2 *cpuid,if (!entry)continue ; -if (tdx_unsupported_cpuid (entry)) +/* +* Reject entries whose index does not match the expected one. +* This catches userspace passing a CPUID entry with the +* KVM_CPUID_FLAG_SIGNIFCANT_INDEX flag cleared when the index +* is significant. +*/ +if (entry->index != tmp.index) +return -EINVAL ; + +if (tdx_has_unsupported_cfg_cpuid_bit (entry))return -EINVAL ; copy_cnt++;
Signature · 1 line
2.46.0
[email protected]
AI reviewRe: [PATCH v3 4/4] KVM: TDX: Validate userspace CPUID input for KVM_TDX_INIT_VM
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing bounds check on the hardware-reported num_cpuid_config leads to an out-of-bounds heap write when initializing TD parameters if the TDX module reports more than 48 CPUID configuration leaves. --
commit 491ef15b15a75d61b2ded06ddb90ff6562206292
Author: Binbin Wu <[email protected]>
KVM: TDX: Validate userspace CPUID input for KVM_TDX_INIT_VM
This commit validates the CPUID configuration provided by userspace through KVM_TDX_INIT_VM against KVM's TDX allowlist. It drops the hardcoded denylist-based check and instead rejects entries matching unsupported CPUID bits or entries with mismatching subleaf indices.
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +0
-0
[ ... ]
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +3
-1@@ -2556,7 +2548,16 @@ static int setup_tdparams_cpuids(struct kvm_cpuid2 *cpuid,if (!entry)continue ; -if (tdx_unsupported_cpuid (entry)) +/* +* Reject entries whose index does not match the expected one. +* This catches userspace passing a CPUID entry with the
arch/x86/kvm/vmx/tdx.c static int setup_tdparams_cpuids(struct kvm_cpuid2 *cpuid, · line 2563
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-0+* KVM_CPUID_FLAG_SIGNIFCANT_INDEX flag cleared when the index +* is significant. +*/ +if (entry->index != tmp.index) +return -EINVAL ; + +if (tdx_has_unsupported_cfg_cpuid_bit (entry))return -EINVAL ; copy_cnt++;
[Severity: High]
This is a pre-existing issue, but does this loop in setup_tdparams_cpuids() lack a bounds check on the hardware-reported td_conf->num_cpuid_config?
Looking at the loop just below this diff in arch/x86/kvm/vmx/tdx.c:
for (i = 0; i < td_conf->num_cpuid_config; i++) {
...
value = &td_params->cpuid_values[i];
value->eax = entry->eax;
...The target td_params is allocated with a strict 1024-byte size bound in tdx_td_init():
td_params = kzalloc_obj(struct td_params);
However, the hardware-reported num_cpuid_config can be up to 128. Because struct td_params contains preceding fields and is bounded to 1024 bytes, the cpuid_values flex array can safely hold a maximum of 48 entries.
If a TDX module supports more than 48 CPUID configuration leaves, this iteration will blindly index past the 48th element, resulting in an out-of-bounds heap write with user-controlled CPUID values.
Should num_cpuid_config be validated against the maximum capacity of the td_params allocation?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH v3 4/4] KVM: TDX: Validate userspace CPUID input for KVM_TDX_INIT_VM
Quoted 61 lines · On 8/27/2026 11:24 AM, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Missing bounds check on the hardware-reportednum_cpuid_configleads to an out-of-bounds heap write when initializing TD parameters if the TDX module reports more than 48 CPUID configuration leaves. --commit
491ef15b15a75d61b2ded06ddb90ff6562206292
Author: Binbin Wu <[email protected]>KVM: TDX: Validate userspace CPUID input for
KVM_TDX_INIT_VMThis commit validates the CPUID configuration provided by userspace through
KVM_TDX_INIT_VMagainst KVM's TDX allowlist. It drops the hardcoded denylist-based check and instead rejects entries matching unsupported CPUID bits or entries with mismatching subleaf indices.Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +0
-0[ ... ]
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +3
-1@@ -2556,7 +2548,16 @@ static int setup_tdparams_cpuids(struct kvm_cpuid2 *cpuid,if (!entry)continue ; -if (tdx_unsupported_cpuid (entry)) +/* +* Reject entries whose index does not match the expected one. +* This catches userspace passing a CPUID entry with the arch/x86/kvm/vmx/tdx.c static int setup_tdparams_cpuids(struct kvm_cpuid2 *cpuid, · line 2563
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-0+* KVM_CPUID_FLAG_SIGNIFCANT_INDEX flag cleared when the index +* is significant. +*/ +if (entry->index != tmp.index) +return -EINVAL ; + +if (tdx_has_unsupported_cfg_cpuid_bit (entry))return -EINVAL ; copy_cnt++;[Severity: High]
This is a pre-existing issue, but does this loop insetup_tdparams_cpuids()lack a bounds check on the hardware-reportedtd_conf->num_cpuid_config?Looking at the loop just below this diff in
arch/x86/kvm/vmx/tdx.c:for (i = 0; i < td_conf->num_cpuid_config; i++) { ... value = &td_params->cpuid_values[i]; value->eax = entry->eax; ...The target
td_paramsis allocated with a strict 1024-byte size bound intdx_td_init():td_params = kzalloc_obj(struct td_params);However, the hardware-reported
num_cpuid_configcan be up to 128. Because structtd_paramscontains preceding fields and is bounded to 1024 bytes, thecpuid_valuesflex array can safely hold a maximum of 48 entries.
Quoted 7 lines
If a TDX module supports more than 48 CPUID configuration leaves, this iteration will blindly index past the 48th element, resulting in an out-of-bounds heap write with user-controlled CPUID values.
Should
num_cpuid_configbe validated against the maximum capacity of thetd_paramsallocation?
td_conf->num_cpuid_config is reported by the TDX module and gives the number of leaves with directly configurable CPUIDs. struct td_params is defined per the TDX spec with a fixed size of 1024 bytes, which is large enough to hold every directly configurable CPUID leaf, i.e. exactly num_cpuid_config entries. So the TDX module should never report a number larger than td_params can accommodate.
So I think the finding is not a real vulnerability, but a cheap one-time sanity check is still defensible.
I will send it as a separate hardening patch if there is no objection.
Edgecombe, Rick P
maintainerQuoted 45 lines · Binbin Wu
Hi,
The purpose of this patch series is to prevent userspace from enabling host state clobbering features that KVM does not support for TDX. A host state clobbering feature exposed on a new TDX module/platform can corrupt host state if KVM does not explicitly save and restore the related MSR(s) across host/guest transitions. If such a feature is blindly exposed to and used by a TD, the host will behave unexpectedly.
Except for a few fixed-1 bits required for basic TDX support, host state clobbering features are either directly configurable or gated by TD ATTRIBUTES/XFAM. So an allowlist covering only the directly configurable CPUID bits, plus the corresponding filtering and validation, is sufficient to serve the purpose while keeping the code footprint small.
Open question for Sean/Paolo
============================
This revision comes long after Sean confirmed the overall direction on RFC v2 [1], due to internal discussions about a potential backward compatibility issue: a CPUID field can change virtualization type to directly configurable. The TDX module can maintain backward compatibility in most cases; the problematic case is a single bit that was effectively "1" and non-configurable becoming directly configurable. The only scenario we can envision for that is an x86 feature deprecation.In that case the TDX module has to decide how to handle deprecation, especially with respect to TD migration compatibility. That pre-existing VMs with the deprecated feature enabled cannot migrate from an old platform to a new one is a problem shared by TDX and VMX, but the TDX module additionally has to decide:
- Whether to allow TDs newly created on old platforms to migrate to new platforms.
- Whether to allow TDs created on new platforms to migrate to old platforms.There are two options:
1. Do not offer the VMM the ability to configure the bit, i.e. simply report it as zero on platforms that no longer support the feature. This disallows both migration cases above.
2. Make the bit directly configurable so the VMM can clear it even on platforms that still support the feature, allowing both migration cases above. However, since the bit is not in KVM's allowlist, userspace would no longer be able to enable it on old platforms after a TDX module update.
Quoted 3 lines
Adding all supported CPUID bits to the allowlist, regardless of their virtualization type in the TDX module, would avoid the compatibility issue for option 2, at the cost of a significantly larger series.
I think we could only do this if no new fixed-1 bits are added. If they were, then a future fixes-1 bit could be added then deprecated. An old kernel would not know about it.
But we will not see new fixed-1 bits?
Quoted 5 lines
Given that x86 feature deprecation is expected to be very rare, and that Intel will discuss such cases with the community beforehand, does the approach taken by this series look acceptable?
In the event a feature bit is turned configurable to support creating TDs that can be migrated to new platforms without the feature, then the desired value for the bit on the old platform will be 0. Which is what it would end up being set as, if the bit was not in the KVM side allow list. So everything will work as desired for that use?
But a user on the old platform that didn't care about migration and was using the feature, would suddenly lose it after TDX module upgrade that turned the bit configurable
But it isn't a kernel bug, so I'd think to punt on this and let TDX module figure out a solution if/when the time comes. Probably various opt-in knobs to allow for working around it. If they are not worried, we don't need to worry and bloat the kernel to prepare for it.
Quoted 18 lines
If so, the goal for this
revision is to collect detailed feedback and, where appropriate, Reviewed-by tags.Expected host state clobbering behavior for TDX
===============================================
We also want to call for discussions about the expected host state clobbering behavior for TDX here for future features.For a normal VMX guest, VM entry/exit behavior for a given piece of CPU state is architecturally defined: state is either switched by hardware via VMCS host/guest fields, or left as the guest value on VM exit and managed by KVM in software.
For TDs, the host/guest transition goes through TDH.VP.ENTER, and what the TDX module does with a given piece of host state is defined by the TDX module ABI rather than by the x86 architecture.
Quoted 7 lines
What we would like to align on is the expected baseline behavior of TDH.VP.ENTER for future features. The proposal is to have TDX simply match VMX behavior, i.e. on return from TDH.VP.ENTER, state that VMX would restore from the VMCS host fields is restored, and state that VMX would leave as the guest value is clobbered. That keeps a single model for VMM, and means enabling a new feature for TDs requires the same work flow as enabling it for VMX.
Ideally the TDX save/restore would share code with normal VMs. On the other hand if we don't share enter/exit paths sufficiently, we may need to duplicate some save/restore in tdx code. Depending on how much of which category we have, it could be better for the kernel to have either one.
And if TDX always saved/restored all state across VP.ENTER, then we don't need bit filtering? What are the problems then with exposing everything to userspace as we currently do?
Quoted 7 lines
FRED is a useful concrete example. Under VMX, the FRED host state in
IA32_FRED_CONFIG,IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX module is expected to restore these MSRs on TDH.VP.ENTER return.IA32_FRED_RSP0andIA32_PL0_SSP(a.k.a.IA32_FRED_SSP0) are handled by software, so the TDX module is expected to clobber them on TDH.VP.ENTER return.
And so the ability to have a design like this for the feature depends on CPUID bit filtering to be in place, otherwise new modules on old kernels can clobber host state unexpectedly. Future features that follow this approach will need some VMM opt-in to say "VMM is filtering CPUID bit config", so TDX can safely expose new clobbering bits. Otherwise, features like FRED will be problems on old kernels. Do we want it? Where would it get plugged in?
Quoted 50 lines
The series
==========Starting with v2 [2], the series takes a simpler, TDX-contained approach instead of the comprehensive CPUID paranoid verification framework across VMX, SVM, and TDX proposed in v1 [3]. It validates only the TDX directly configurable CPUID bits, which are reported by the TDX module in
CPUID_CONFIGfields that the VMM can configure for a TD. All filtering and validation logic is isolated within TDX code. This is sufficient to address the host clobbering issue because:
- The TDX module will not introduce new host state clobbering features that are fixed-1.
- Non-directly configurable feature bits (i.e. features controlled by XFAM or ATTRIBUTES) cannot be enabled by userspace without KVM's support.Specifically, this series builds a KVM-side allowlist of supported TDX directly configurable CPUID bits to:
- FilterKVM_TDX_CAPABILITIES
Replace the hardcoded denylist to only report configurable bits that KVM explicitly supports.
- ValidateKVM_TDX_INIT_VM
Reject any configurable bit that the TDX module allows but KVM does not support, as well as CPUID entries with an unexpected subleaf.With this allowlist, newly added TDX configurable CPUID bits will not be exposed to userspace until KVM explicitly opts in after fulfilling the necessary virtualization requirements.
The allowlist consists of two parts:
- Feature CPUID bits, which are tracked in tdx_cpu_cfg_caps[] following the organization of kvm_cpu_caps[]. It holds KVM's supported masks for the TDX configurable CPUID feature bits.
- Non-feature multi-bit fields, which are handled at runtime for the CPUID registers holding such fields.For compatibility with older TDX modules that report
CORE_CAPABILITIESas fixed-1, reportCORE_CAPABILITIESas configurable even though KVM does not support guest access to MSR_IA32_CORE_CAPS. This allows userspace to keep enablingCORE_CAPABILITIESwhen the bit changes from fixed-1 to directly configurable, and allows userspace to infer that the bit is no longer fixed-1 so it can adjust its expectations.Known limitation:
- The series does not checkKVM_SET_CPUID2input for consistency with the CPUID configuration supplied through KVM_TDX_INIT_VM. This is not required for host safety today because KVM does not use its vCPU CPUID model to decide whether to manage host-clobbering state for TDs. A consistency check should be added if KVM ever starts relying on the vCPU CPUID model for such decisions.
Quoted 50 lines · Rick P
Quoted 45 lines · Binbin Wu
Hi,
The purpose of this patch series is to prevent userspace from enabling host state clobbering features that KVM does not support for TDX. A host state clobbering feature exposed on a new TDX module/platform can corrupt host state if KVM does not explicitly save and restore the related MSR(s) across host/guest transitions. If such a feature is blindly exposed to and used by a TD, the host will behave unexpectedly.
Except for a few fixed-1 bits required for basic TDX support, host state clobbering features are either directly configurable or gated by TD ATTRIBUTES/XFAM. So an allowlist covering only the directly configurable CPUID bits, plus the corresponding filtering and validation, is sufficient to serve the purpose while keeping the code footprint small.
Open question for Sean/Paolo
============================
This revision comes long after Sean confirmed the overall direction on RFC v2 [1], due to internal discussions about a potential backward compatibility issue: a CPUID field can change virtualization type to directly configurable. The TDX module can maintain backward compatibility in most cases; the problematic case is a single bit that was effectively "1" and non-configurable becoming directly configurable. The only scenario we can envision for that is an x86 feature deprecation.In that case the TDX module has to decide how to handle deprecation, especially with respect to TD migration compatibility. That pre-existing VMs with the deprecated feature enabled cannot migrate from an old platform to a new one is a problem shared by TDX and VMX, but the TDX module additionally has to decide:
- Whether to allow TDs newly created on old platforms to migrate to new platforms.
- Whether to allow TDs created on new platforms to migrate to old platforms.There are two options:
1. Do not offer the VMM the ability to configure the bit, i.e. simply report it as zero on platforms that no longer support the feature. This disallows both migration cases above.
2. Make the bit directly configurable so the VMM can clear it even on platforms that still support the feature, allowing both migration cases above. However, since the bit is not in KVM's allowlist, userspace would no longer be able to enable it on old platforms after a TDX module update.Quoted 3 lines
Adding all supported CPUID bits to the allowlist, regardless of their virtualization type in the TDX module, would avoid the compatibility issue for option 2, at the cost of a significantly larger series.
Quoted 5 lines
I think we could only do this if no new fixed-1 bits are added. If they were, then a future fixes-1 bit could be added then deprecated. An old kernel would not know about it.
But we will not see new fixed-1 bits?
The expectation is that no new fixed-1 bits will be added after the initial TDX basic support. A new fixed-1 bit means the x86 architecture itself leaves the TDX module no choice, so the TDX module will not add one unless some very unusual future architectural change forces it to (and it's hard to see such a design passing the design review in practice). If that ever does happen, it would imply a significant architectural change, and we can figure out how to handle it then.
Quoted 7 lines
Quoted 6 lines
Given that x86 feature deprecation is expected to be very rare, and that Intel will discuss such cases with the community beforehand, does the approach taken by this series look acceptable?
Quoted 5 lines
In the event a feature bit is turned configurable to support creating TDs that can be migrated to new platforms without the feature, then the desired value for the bit on the old platform will be 0. Which is what it would end up being set as, if the bit was not in the KVM side allow list. So everything will work as desired for that use?
Yes, it would work for migration cases.
Quoted 4 lines
But a user on the old platform that didn't care about migration and was using the feature, would suddenly lose it after TDX module upgrade that turned the bit configurable
Yes. This is exactly the backward compatibility issue for option 2. Though I am not sure how important it is when losing a feature that is being deprecated.
Quoted 4 lines
But it isn't a kernel bug, so I'd think to punt on this and let TDX module figure out a solution if/when the time comes. Probably various opt-in knobs to allow for working around it.
Do you mean on the old platforms, the according TDX module versions add opt-in knobs to allow the "effective 1 -> directly configurable" change?
Quoted 29 lines
If they are not worried, we don't need to worry and bloat the kernel to prepare for it.
Quoted 18 lines
If so, the goal for this
revision is to collect detailed feedback and, where appropriate, Reviewed-by tags.Expected host state clobbering behavior for TDX
===============================================
We also want to call for discussions about the expected host state clobbering behavior for TDX here for future features.For a normal VMX guest, VM entry/exit behavior for a given piece of CPU state is architecturally defined: state is either switched by hardware via VMCS host/guest fields, or left as the guest value on VM exit and managed by KVM in software.
For TDs, the host/guest transition goes through TDH.VP.ENTER, and what the TDX module does with a given piece of host state is defined by the TDX module ABI rather than by the x86 architecture.
Quoted 7 lines
What we would like to align on is the expected baseline behavior of TDH.VP.ENTER for future features. The proposal is to have TDX simply match VMX behavior, i.e. on return from TDH.VP.ENTER, state that VMX would restore from the VMCS host fields is restored, and state that VMX would leave as the guest value is clobbered. That keeps a single model for VMM, and means enabling a new feature for TDs requires the same work flow as enabling it for VMX.
Quoted 3 lines
Ideally the TDX save/restore would share code with normal VMs. On the other hand if we don't share enter/exit paths sufficiently, we may need to duplicate some save/restore in tdx code.
It probably needs some TDX specific handling, since the TDX module clobbers the MSRs (setting them to either their INIT values or some default values), whereas in the VMX case the guest values are left.
Depending on how much of which category we have, it
Quoted 5 lines
could be better for the kernel to have either one.
And if TDX always saved/restored all state across VP.ENTER, then we don't need bit filtering? What are the problems then with exposing everything to userspace as we currently do?
One consideration is performance.
- When The TDX module clobbers an MSR, there are 1 RDMSR + 2 WRMSR TDX WRMSR to restore guest value
---
TDX RDMSR to save guest value
TDX WRMSR to set to default value
- When The TDX module save/restore an MSR, there are 2 RDMSR + 2 WRMSR TDX RDMSR to save host value
TDX WRMSR to restore guest value
---
TDX RDMSR to save guest value
TDX WRMSR to restore host value
In most cases where the TDX module chooses to clobber an MSR rather than restore the host value, that MSR isn't consumed in ring 0, so the Linux kernel/KVM simply restores the value before returning to userspace.
That means for most TD entry/exit paths there is one extra MSR read per MSR.
Previously in the PUCK meeting, we proposed that the TDX module preserve host state by default for any new feature, and provide an interface for the host VMM to opt in to "don't preserve" as an optimization. That proposal wasn't pursued, since Sean suggested letting KVM do the CPUID validation for TDX instead.
Quoted 9 lines
Quoted 8 lines
FRED is a useful concrete example. Under VMX, the FRED host state in
IA32_FRED_CONFIG,IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX module is expected to restore these MSRs on TDH.VP.ENTER return.IA32_FRED_RSP0andIA32_PL0_SSP(a.k.a.IA32_FRED_SSP0) are handled by software, so the TDX module is expected to clobber them on TDH.VP.ENTER return.
Quoted 3 lines
And so the ability to have a design like this for the feature depends on CPUID bit filtering to be in place, otherwise new modules on old kernels can clobber host state unexpectedly.
Yes.
Quoted 4 lines
Future features that follow this approach will need
some VMM opt-in to say "VMM is filtering CPUID bit config", so TDX can safely expose new clobbering bits. Otherwise, features like FRED will be problems on old kernels.
If FRED support lands in KVM for non-TDX VMs before this filtering/invalidation is in place, adding FRED to the hardcoded denylist would work as a temporary solution.
But that only holds for a well-behaved userspace VMM. QEMU, for example, only enables features supported by both KVM and TDX, so if FRED isn't supported in KVM for non-TDX VMs, QEMU won't enable it for TDX either.
A malicious userspace VMM, however, can set a feature regardless of KVM's reported CPU capabilities, and could still cause trouble.
Quoted 1 line
Do we want it? Where would it get plugged in?
So if the approach in this series is taken, I think we still need such an opt-in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Edgecombe, Rick P
maintainerQuoted 7 lines · Binbin Wu
Quoted 4 lines
But it isn't a kernel bug, so I'd think to punt on this and let TDX module figure out a solution if/when the time comes. Probably various opt-in knobs to allow for working around it.
Do you mean on the old platforms, the according TDX module versions add opt-in knobs to allow the "effective 1 -> directly configurable" change?
Yea. Or the other way. Forcing the old fixed-1 bits to 1 automatically. Could be like a quirk like config thing. User/admin can decide if they want migration flexible design, or backwards compatibility for existing TDs. BUT, we could work out the details later if we think we have options.
I think I've convinced myself we have options and can close this one. Agreed?
<snip>
Quoted 5 lines
If FRED support lands in KVM for non-TDX VMs before this filtering/invalidation is in place, adding FRED to the hardcoded denylist would work as a temporary solution.
This could easily happen.
Quoted 7 lines
But that only holds for a well-behaved userspace VMM. QEMU, for example, only enables features supported by both KVM and TDX, so if FRED isn't supported in KVM for non-TDX VMs, QEMU won't enable it for TDX either.
A malicious userspace VMM, however, can set a feature regardless of KVM's reported CPU capabilities, and could still cause trouble.
Right. It the problem I thought we would need an opt-in to avoid.
Quoted 7 lines
Quoted 2 lines
Do we want it? Where would it get plugged in?
So if the approach in this series is taken, I think we still need such an opt- in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Yep. Can we think about what it would look like? Easiest would be a bit passed in TDH_SYS_CONFIG. But then arch/x86 is saying how KVM will behave. Ok to me, for the simplicity. Could come with a nice comment.
Quoted 6 lines · Rick P
Quoted 5 lines · Binbin Wu
Quoted 4 lines
But it isn't a kernel bug, so I'd think to punt on this and let TDX module figure out a solution if/when the time comes. Probably various opt-in knobs to allow for working around it.
Quoted 6 lines
Quoted 2 lines
Do you mean on the old platforms, the according TDX module versions add opt-in knobs to allow the "effective 1 -> directly configurable" change?
Yea. Or the other way. Forcing the old fixed-1 bits to 1 automatically. Could be like a quirk like config thing. User/admin can decide if they want migration flexible design, or backwards compatibility for existing TDs.
It's all about the new created TDs on the old platforms. For existing TDs, the runtime TDX module update should not change the shape of the TDs.
Quoted 4 lines
BUT, we could work
out the details later if we think we have options.I think I've convinced myself we have options and can close this one. Agreed?
Agree.
Quoted 23 lines
<snip>
Quoted 5 lines
If FRED support lands in KVM for non-TDX VMs before this filtering/invalidation is in place, adding FRED to the hardcoded denylist would work as a temporary solution.
This could easily happen.
Quoted 7 lines
But that only holds for a well-behaved userspace VMM. QEMU, for example, only enables features supported by both KVM and TDX, so if FRED isn't supported in KVM for non-TDX VMs, QEMU won't enable it for TDX either.
A malicious userspace VMM, however, can set a feature regardless of KVM's reported CPU capabilities, and could still cause trouble.
Right. It the problem I thought we would need an opt-in to avoid.
Quoted 3 lines
Quoted 2 lines
Do we want it? Where would it get plugged in?
Quoted 8 lines
Quoted 4 lines
So if the approach in this series is taken, I think we still need such an opt- in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Yep. Can we think about what it would look like? Easiest would be a bit passed in
TDH_SYS_CONFIG. But thenarch/x86is saying how KVM will behave. Ok to me, for the simplicity. Could come with a nice comment.
The TDX module is initialized before KVM is loaded. I guess the upstream kernel doesn't support out of tree KVM code, so we can assume if the kernel has the code to opt-in the new host state clobbering features, KVM must have implemented the TDX CPUID filtering and validation?
Also, do you think it's reasonable to backport this patch series to stable/LTS kernels as an alternative?
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Binbin WuQuoted 30 lines · Binbin Wu
arch/x86/kvm/vmx/tdx.c +24
-0@@ -52,6 +52,149 @@ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #a3 " 0x%llx", \ a1, a2, a3) +static u32 tdx_cpu_cfg_caps[NR_KVM_CPU_CAPS ]__ro_after_init ; +static_assert (ARRAY_SIZE (tdx_cpu_cfg_caps) ==ARRAY_SIZE (kvm_cpu_caps)); + +#define TDX_VALIDATE_CPU_CAP_USAGE (name) \ + BUILD_BUG_ON(__feature_leaf(X86_FEATURE_##name) != \ + tdx_cpu_cap_init_in_progress) + +/* For feature bit that KVM advertised through kvm_cpu_caps[]. */ +#define TDX_CFG_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_caps |= feature_bit(name); \ +}) + +/* +* For feature bit KVM allows for TDX guests even though it is not advertised +* through kvm_cpu_caps[], e.g. MWAIT. +*/ +#define TDX_CFG_EXTRA_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_extra_caps |= feature_bit(name); \ +}) +
arch/x86/kvm/vmx/tdx.c line 88
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+#define tdx_cpu_cfg_cap_init(leaf, feature_initializers...) \ +do { \ + const u32 __maybe_unused tdx_cpu_cap_init_in_progress = leaf; \ + u32 tdx_cfg_extra_caps = 0; \ + u32 tdx_cfg_caps = 0; \ + \ + feature_initializers \ + tdx_cpu_cfg_caps[leaf] = (tdx_cfg_caps & kvm_cpu_caps[leaf]) | \ + tdx_cfg_extra_caps; \ +} while (0)
How about make some of the above into just static functions for easier readabilyt?
And then drop the __maybe_unused for tdx_cpu_cap_init_in_progress?
Re: [PATCH v3 3/4] KVM: TDX: Filter configurable CPUID bits
In reply to Binbin WuQuoted 2 lines · Binbin Wu
arch/x86/kvm/vmx/tdx.c +0
-0
...
Quoted 20 lines
arch/x86/kvm/vmx/tdx.c +17
-2+static u32 tdx_cfg_non_feature_mask (u32 function,u32 index,int reg) { -if (has_tsx (entry)) -clear_tsx (entry); +/* +* For a leaf/subleaf/register that will never be repurposed to hold +* feature bits, it's safe to return TDX_CPUID_ALL_ALLOWED_MASK, i.e. +* leave the TDX module's CPUID config mask intact. +*/ +switch (function) { +case 1 : +if (reg ==CPUID_EAX || reg ==CPUID_EBX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +case 4 : +case 0x18 : +case 0x1f : +return TDX_CPUID_ALL_ALLOWED_MASK ; +case 0x24 : +if (index ==0 && reg ==CPUID_EBX )
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 320
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+return GENMASK_U32 (7 ,0 ); +return 0 ; +case 0x80000008 : +if (reg ==CPUID_EAX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +default : +return 0 ; + } +}
How about rename the above to something simpler like tdx_get_cpuid_bits()? Sorry I don't have anything better to suggest for naming.
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+static u32 tdx_cfg_feature_mask (u32 function,u32 index,int reg) +{ +for (int i =0 ; i <NR_KVM_CPU_CAPS ; i++) { +const struct cpuid_reg *cpuid = &reverse_cpuid[i]; +
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 336
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +9
-0+if (!cpuid->function) +continue ; + +if (cpuid->function == function && cpuid->index == index && + cpuid->reg == reg) +return tdx_cpu_cfg_caps[i]; + } + +return 0 ; }
And then the above to tdx_get_cpuid_feature_bits()?
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +2
-2-static bool tdx_unsupported_cpuid (const struct kvm_cpuid_entry2 *entry) +static u32 tdx_get_allowed_cfg_cpuid_mask (u32 function,u32 index,int reg) { -return has_tsx (entry) ||has_waitpkg (entry); +u32 non_feature_mask =tdx_cfg_non_feature_mask (function, index, reg);
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 350
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +9
-0+ +if (non_feature_mask ==TDX_CPUID_ALL_ALLOWED_MASK ) +return TDX_CPUID_ALL_ALLOWED_MASK ; + +/* +* It's possible that a CPUID register contains both feature and +* non-feature bits. +*/ +return non_feature_mask |tdx_cfg_feature_mask (function, index, reg); }
And then tdx_get_cpuid_mask()?
Re: [PATCH v3 2/4] KVM: TDX: Report CORE_CAPABILITIES as configurable
In reply to Binbin WuQuoted 11 lines · Binbin Wu
Add
CORE_CAPABILITIES(CPUID.0x7.0.EDX[30]) to KVM's allowlist of TDX directly configurable CPUID feature bits, even though KVM doesn't supportMSR_IA32_CORE_CAPSfor TDX guests, to accommodate legacy TDX module behavior.Older TDX specs define the
CORE_CAPABILITIESCPUID bit as fixed-1, so userspace may expect the bit to be enabled for TDs. If the bit becomes directly configurable in a newer TDX module but is not reported as such to userspace, userspace can no longer enable it once KVM starts validating the CPUID configuration input.
Quoted 8 lines
Reporting
CORE_CAPABILITIESas configurable keeps userspace able to enable the bit across the fixed-1 => configurable transition, and lets userspace infer that the bit is no longer fixed-1 so it can adjust its expectations.Keep
MSR_IA32_CORE_CAPSunsupported for TDX guests, as existing TDX users have not needed guest access to the MSR, and advertising the CPUID bit as configurable is enough for userspace to handle the legacy-module compatibility case.
Re: [PATCH v3 4/4] KVM: TDX: Validate userspace CPUID input for KVM_TDX_INIT_VM
In reply to Binbin WuQuoted 19 lines · Binbin Wu
Validate the CPUID configuration provided by userspace through
KVM_TDX_INIT_VMagainst KVM's TDX allowlist, and drop the hardcoded denylist based check.The TDX module lets the VMM configure certain CPUID features for a TD at initialization time, but KVM must strictly govern which of them userspace can actually enable, otherwise a host state clobbering feature could be enabled behind KVM's back. The existing check only rejects TSX and WAITPKG, i.e. it is not fail-safe, as any bit that a future TDX module makes configurable would be accepted even if KVM has no idea about the feature.
Add
tdx_has_unsupported_cfg_cpuid_bit()and rejectKVM_TDX_INIT_VMif userspace sets any bit outside the mask returned bytdx_get_allowed_cfg_cpuid_mask(). There is no need to first mask the userspace input with the bits the TDX module reports as directly configurable, as anything outside that set is rejected by the TDX module itself.
Quoted 4 lines
Also reject CPUID entries whose index differs from the value expected by the TDX module, as
kvm_find_cpuid_entry2()ignores the index whenKVM_CPUID_FLAG_SIGNIFCANT_INDEXis cleared, and so a mismatching entry could otherwise be applied to the wrong subleaf.
This nicely cleans up the earlier handling:
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Tony LindgrenQuoted 36 lines · Tony Lindgren
Quoted 35 lines · Binbin Wu
arch/x86/kvm/vmx/tdx.c +29
-0@@ -52,6 +52,149 @@ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #a3 " 0x%llx", \ a1, a2, a3) +static u32 tdx_cpu_cfg_caps[NR_KVM_CPU_CAPS ]__ro_after_init ; +static_assert (ARRAY_SIZE (tdx_cpu_cfg_caps) ==ARRAY_SIZE (kvm_cpu_caps)); + +#define TDX_VALIDATE_CPU_CAP_USAGE (name) \ + BUILD_BUG_ON(__feature_leaf(X86_FEATURE_##name) != \ + tdx_cpu_cap_init_in_progress) + +/* For feature bit that KVM advertised through kvm_cpu_caps[]. */ +#define TDX_CFG_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_caps |= feature_bit(name); \ +}) + +/* +* For feature bit KVM allows for TDX guests even though it is not advertised +* through kvm_cpu_caps[], e.g. MWAIT. +*/ +#define TDX_CFG_EXTRA_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_extra_caps |= feature_bit(name); \ +}) + +#define tdx_cpu_cfg_cap_init(leaf, feature_initializers...) \ +do { \ + const u32 __maybe_unused tdx_cpu_cap_init_in_progress = leaf; \ + u32 tdx_cfg_extra_caps = 0; \ + u32 tdx_cfg_caps = 0; \
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c line 88
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+ \ +feature_initializers \ + tdx_cpu_cfg_caps[leaf] = (tdx_cfg_caps & kvm_cpu_caps[leaf]) | \ + tdx_cfg_extra_caps; \ +}while (0 )How about make some of the above into just static functions for easier readabilyt?
Thanks for your review!
This deliberately mirrors the existing kvm_cpu_cap_init() / F() / XXX_F() pattern in arch/x86/kvm/cpuid.c.
Similar to kvm_cpu_cap_init(), tdx_cpu_cfg_cap_init() takes a variadic list of statements that mutate two local accumulators (tdx_cfg_caps, tdx_cfg_extra_caps). That's inherently a macro-scoped construct.
Quoted 2 lines
And then drop the
__maybe_unusedfortdx_cpu_cap_init_in_progress?
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
Quoted 46 lines · Binbin Wu
Quoted 36 lines · Tony Lindgren
Quoted 35 lines · Binbin Wu
arch/x86/kvm/vmx/tdx.c +29
-0@@ -52,6 +52,149 @@ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #a3 " 0x%llx", \ a1, a2, a3) +static u32 tdx_cpu_cfg_caps[NR_KVM_CPU_CAPS ]__ro_after_init ; +static_assert (ARRAY_SIZE (tdx_cpu_cfg_caps) ==ARRAY_SIZE (kvm_cpu_caps)); + +#define TDX_VALIDATE_CPU_CAP_USAGE (name) \ + BUILD_BUG_ON(__feature_leaf(X86_FEATURE_##name) != \ + tdx_cpu_cap_init_in_progress) + +/* For feature bit that KVM advertised through kvm_cpu_caps[]. */ +#define TDX_CFG_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_caps |= feature_bit(name); \ +}) + +/* +* For feature bit KVM allows for TDX guests even though it is not advertised +* through kvm_cpu_caps[], e.g. MWAIT. +*/ +#define TDX_CFG_EXTRA_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_extra_caps |= feature_bit(name); \ +}) + +#define tdx_cpu_cfg_cap_init(leaf, feature_initializers...) \ +do { \ + const u32 __maybe_unused tdx_cpu_cap_init_in_progress = leaf; \ + u32 tdx_cfg_extra_caps = 0; \ + u32 tdx_cfg_caps = 0; \Quoted 8 lines
arch/x86/kvm/vmx/tdx.c line 88
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+ \ +feature_initializers \ + tdx_cpu_cfg_caps[leaf] = (tdx_cfg_caps & kvm_cpu_caps[leaf]) | \ + tdx_cfg_extra_caps; \ +}while (0 )How about make some of the above into just static functions for easier readabilyt?
Quoted 8 lines
Thanks for your review!
This deliberately mirrors the existing
kvm_cpu_cap_init()/F()/XXX_F()pattern inarch/x86/kvm/cpuid.c.Similar to
kvm_cpu_cap_init(),tdx_cpu_cfg_cap_init()takes a variadic list of statements that mutate two local accumulators (tdx_cfg_caps,tdx_cfg_extra_caps). That's inherently a macro-scoped construct.
Ah OK thanks I now noticed the feature_initializers.
Re: [PATCH v3 3/4] KVM: TDX: Filter configurable CPUID bits
In reply to Tony LindgrenQuoted 65 lines · Tony Lindgren
Quoted 2 lines · Binbin Wu
arch/x86/kvm/vmx/tdx.c +0
-0...
Quoted 20 lines
arch/x86/kvm/vmx/tdx.c +17
-2+static u32 tdx_cfg_non_feature_mask (u32 function,u32 index,int reg) { -if (has_tsx (entry)) -clear_tsx (entry); +/* +* For a leaf/subleaf/register that will never be repurposed to hold +* feature bits, it's safe to return TDX_CPUID_ALL_ALLOWED_MASK, i.e. +* leave the TDX module's CPUID config mask intact. +*/ +switch (function) { +case 1 : +if (reg ==CPUID_EAX || reg ==CPUID_EBX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +case 4 : +case 0x18 : +case 0x1f : +return TDX_CPUID_ALL_ALLOWED_MASK ; +case 0x24 : +if (index ==0 && reg ==CPUID_EBX )arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 320
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+return GENMASK_U32 (7 ,0 ); +return 0 ; +case 0x80000008 : +if (reg ==CPUID_EAX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +default : +return 0 ; + } +}How about rename the above to something simpler like
tdx_get_cpuid_bits()? Sorry I don't have anything better to suggest for naming.Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+static u32 tdx_cfg_feature_mask (u32 function,u32 index,int reg) +{ +for (int i =0 ; i <NR_KVM_CPU_CAPS ; i++) { +const struct cpuid_reg *cpuid = &reverse_cpuid[i]; +arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 336
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +9
-0+if (!cpuid->function) +continue ; + +if (cpuid->function == function && cpuid->index == index && + cpuid->reg == reg) +return tdx_cpu_cfg_caps[i]; + } + +return 0 ; }And then the above to
tdx_get_cpuid_feature_bits()?Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +6
-2-static bool tdx_unsupported_cpuid (const struct kvm_cpuid_entry2 *entry) +static u32 tdx_get_allowed_cfg_cpuid_mask (u32 function,u32 index,int reg) { -return has_tsx (entry) ||has_waitpkg (entry); +u32 non_feature_mask =tdx_cfg_non_feature_mask (function, index, reg); + +if (non_feature_mask ==TDX_CPUID_ALL_ALLOWED_MASK ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 350
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +5
-0+/* +* It's possible that a CPUID register contains both feature and +* non-feature bits. +*/ +return non_feature_mask |tdx_cfg_feature_mask (function, index, reg); }And then
tdx_get_cpuid_mask()?
How about:tdx_get_cpuid_cfg_non_feature_mask()tdx_get_cpuid_cfg_feature_mask()tdx_get_cpuid_cfg_mask()
Re: [PATCH v3 3/4] KVM: TDX: Filter configurable CPUID bits
Quoted 75 lines · Binbin Wu
Quoted 65 lines · Tony Lindgren
Quoted 2 lines · Binbin Wu
arch/x86/kvm/vmx/tdx.c +0
-0...
Quoted 20 lines
arch/x86/kvm/vmx/tdx.c +17
-2+static u32 tdx_cfg_non_feature_mask (u32 function,u32 index,int reg) { -if (has_tsx (entry)) -clear_tsx (entry); +/* +* For a leaf/subleaf/register that will never be repurposed to hold +* feature bits, it's safe to return TDX_CPUID_ALL_ALLOWED_MASK, i.e. +* leave the TDX module's CPUID config mask intact. +*/ +switch (function) { +case 1 : +if (reg ==CPUID_EAX || reg ==CPUID_EBX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +case 4 : +case 0x18 : +case 0x1f : +return TDX_CPUID_ALL_ALLOWED_MASK ; +case 0x24 : +if (index ==0 && reg ==CPUID_EBX )arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 320
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+return GENMASK_U32 (7 ,0 ); +return 0 ; +case 0x80000008 : +if (reg ==CPUID_EAX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +default : +return 0 ; + } +}How about rename the above to something simpler like
tdx_get_cpuid_bits()? Sorry I don't have anything better to suggest for naming.Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5
-0+static u32 tdx_cfg_feature_mask (u32 function,u32 index,int reg) +{ +for (int i =0 ; i <NR_KVM_CPU_CAPS ; i++) { +const struct cpuid_reg *cpuid = &reverse_cpuid[i]; +arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 336
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +9
-0+if (!cpuid->function) +continue ; + +if (cpuid->function == function && cpuid->index == index && + cpuid->reg == reg) +return tdx_cpu_cfg_caps[i]; + } + +return 0 ; }And then the above to
tdx_get_cpuid_feature_bits()?Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +6
-2-static bool tdx_unsupported_cpuid (const struct kvm_cpuid_entry2 *entry) +static u32 tdx_get_allowed_cfg_cpuid_mask (u32 function,u32 index,int reg) { -return has_tsx (entry) ||has_waitpkg (entry); +u32 non_feature_mask =tdx_cfg_non_feature_mask (function, index, reg); + +if (non_feature_mask ==TDX_CPUID_ALL_ALLOWED_MASK ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +Quoted 8 lines
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 350
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +5
-0+/* +* It's possible that a CPUID register contains both feature and +* non-feature bits. +*/ +return non_feature_mask |tdx_cfg_feature_mask (function, index, reg); }And then
tdx_get_cpuid_mask()?
Quoted 4 lines
How about:
tdx_get_cpuid_cfg_non_feature_mask()tdx_get_cpuid_cfg_feature_mask()tdx_get_cpuid_cfg_mask()
I'd be happy with that.
Quoted 2 lines · Binbin Wu
Quoted 1 line · Rick P
On Thu, 2026-08-27 at 11:18 +0800, Binbin Wu wrote:
<snip>
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Expected host state clobbering behavior for TDX
===============================================
We also want to call for discussions about the expected host state clobbering behavior for TDX here for future features.For a normal VMX guest, VM entry/exit behavior for a given piece of CPU state is architecturally defined: state is either switched by hardware via VMCS host/guest fields, or left as the guest value on VM exit and managed by KVM in software.
For TDs, the host/guest transition goes through TDH.VP.ENTER, and what the TDX module does with a given piece of host state is defined by the TDX module ABI rather than by the x86 architecture.
What we would like to align on is the expected baseline behavior of TDH.VP.ENTER for future features. The proposal is to have TDX simply match VMX behavior, i.e. on return from TDH.VP.ENTER, state that VMX would restore from the VMCS host fields is restored, and state that VMX would leave as the guest value is clobbered. That keeps a single model for VMM, and means enabling a new feature for TDs requires the same work flow as enabling it for VMX.
Quoted 3 lines
Quoted 3 lines
Ideally the TDX save/restore would share code with normal VMs. On the other hand if we don't share enter/exit paths sufficiently, we may need to duplicate some save/restore in tdx code.
(Copy the FRED example here for reference)
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
FRED is a useful concrete example. Under VMX, the FRED host state in
IA32_FRED_CONFIG,IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX module is expected to restore these MSRs on TDH.VP.ENTER return.IA32_FRED_RSP0andIA32_PL0_SSP(a.k.a.IA32_FRED_SSP0) are handled by software, so the TDX module is expected to clobber them on TDH.VP.ENTER return.
What I get, is not matching VMX behavior but matching the behavior KVM will perform for VMX. They are based on the assumption that KVM will always enable the save/restore VMCS fields for a new feature. But I don't think we can guarantee it.
To me, "have TDX simply match VMX behavior" means:
1. if the VMX unconditionally save/restore a state, then TDX will do so.
2. if there are vm-entry/vm-exit load/save VMCS fields for a state, then provide the equivalent per-TD configurable interfaces which matches the VMCS fields.
Quoted 10 lines
It probably needs some TDX specific handling, since the TDX module clobbers the MSRs (setting them to either their INIT values or some default values), whereas in the VMX case the guest values are left.
Quoted 6 lines
Depending on how much of which category we have, it could be better for the kernel to have either one.
And if TDX always saved/restored all state across VP.ENTER, then we don't need bit filtering? What are the problems then with exposing everything to userspace as we currently do?
IMO, allowing userspace to expose/enable a new feature to a guest without KVM first evaluating it is always dangerous. It's not just about the state clobbering. We can know the implication for a new feature.
For example, a feature consumes global per-socket resources. Allowing guest to use the feature might slowdown the host.
Another example is a feature is used to catch bad behaviors, and in this case host would like to enforce the feature being forced on for the guest instead of allowing the guest to use (disable) the feature freely.
Quoted 7 lines · Binbin Wu
Quoted 7 lines
Quoted 4 lines
So if the approach in this series is taken, I think we still need such an opt- in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Yep. Can we think about what it would look like? Easiest would be a bit passed in
TDH_SYS_CONFIG. But thenarch/x86is saying how KVM will behave. Ok to me, for the simplicity. Could come with a nice comment.
Can you just treat current KVM behavior of allowing userspace to enable any configurable bits as the bug of KVM and backport this series as Binbin suggested below? Instead of introducing more opt-in knobs.
Quoted 7 lines
The TDX module is initialized before KVM is loaded. I guess the upstream kernel doesn't support out of tree KVM code, so we can assume if the kernel has the code to opt-in the new host state clobbering features, KVM must have implemented the TDX CPUID filtering and validation?
Also, do you think it's reasonable to backport this patch series to stable/LTS kernels as an alternative?
Quoted 10 lines · Xiaoyao Li
Quoted 7 lines · Binbin Wu
Quoted 7 lines
Quoted 4 lines
So if the approach in this series is taken, I think we still need such an opt- in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Yep. Can we think about what it would look like? Easiest would be a bit passed in
TDH_SYS_CONFIG. But thenarch/x86is saying how KVM will behave. Ok to me, for the simplicity. Could come with a nice comment.Can you just treat current KVM behavior of allowing userspace to enable
well, I meant "can we"...
Quoted 10 lines
any configurable bits as the bug of KVM and backport this series as Binbin suggested below? Instead of introducing more opt-in knobs.
Quoted 7 lines
The TDX module is initialized before KVM is loaded. I guess the upstream kernel doesn't support out of tree KVM code, so we can assume if the kernel has the code to opt-in the new host state clobbering features, KVM must have implemented the TDX CPUID filtering and validation?
Also, do you think it's reasonable to backport this patch series to stable/LTS kernels as an alternative?
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Binbin WuQuoted 16 lines · Binbin Wu
Add tdx_cpu_cfg_caps[] to track the subset of TDX directly configurable CPUID feature bits that KVM supports, and build the masks during TDX hardware setup via
tdx_initialize_cpu_cfg_caps().The TDX module reports the CPUID bits that the VMM can directly configure for a TD, but KVM cannot blindly expose all reported bits to userspace. Certain features imply additional architectural state, e.g. one or more MSRs, that KVM must explicitly manage across host/guest transitions to prevent host state corruption.
Today KVM relies on a hardcoded denylist, i.e. it clears a few known problematic bits, e.g. TSX and WAITPKG, and passes everything else through. A denylist is fundamentally fragile while an allowlist inverts the default, i.e. unknown configurable bits are hidden and not allowed to be enabled until KVM explicitly opts in.
Quoted 7 lines
Except for a few fixed-1 bits required for basic TDX support, host state clobbering features are either directly configurable or gated by TD ATTRIBUTES/XFAM.
Tracking only the directly configurable feature bits is therefore sufficient to serve the purpose while keeping the code footprint small.
I'm not clear how it is therefore sufficient. We at least need to explain that ATTRIBUTS/XFAM are validated separately by KVM already?
Quoted 33 lines
Organize tdx_cpu_cfg_caps[] following kvm_cpu_caps[] so that the masks can be built with the similar feature-name based initializers. CPUID registers that hold directly configurable non-feature (multi-bit) fields are handled separately.
The allowlist is consumed by later patches to filter
KVM_TDX_CAPABILITIESand to reject unsupported CPUID input toKVM_TDX_INIT_VM, so that newly introduced TDX directly configurable CPUID feature bits stay hidden from userspace until KVM explicitly opts in.Add comments as placeholders for HLE, RTM and WAITPKG, which KVM doesn't support for TDX yet.
Signed-off-by Binbin Wu <[email protected]>---
v3:
- Drop the new data structure in v2 and only track feature bits by following the organization of kvm_cpu_caps[], handle non-feature bits separately. (Sean)
- Use two versions of macros (TDX_CFG_F()VS.TDX_CFG_EXTRA_F()) to distinguish whether a supported TDX configurable CPUID bit should be checked against KVM's common cpu capabilities.
- AddAMX_COMPLEXsince it has been defined in the CPUID virtualization doc.--- arch/x86/kvm/vmx/tdx.c | 145 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 145 insertions(+)arch/x86/kvm/vmx/tdx.c +0
-0@@ -52,6 +52,149 @@ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #a3 " 0x%llx", \
arch/x86/kvm/vmx/tdx.c line 62
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +8
-0a1, a2, a3) +static u32 tdx_cpu_cfg_caps[NR_KVM_CPU_CAPS ]__ro_after_init ; +static_assert (ARRAY_SIZE (tdx_cpu_cfg_caps) ==ARRAY_SIZE (kvm_cpu_caps)); + +#define TDX_VALIDATE_CPU_CAP_USAGE (name) \ + BUILD_BUG_ON(__feature_leaf(X86_FEATURE_##name) != \ + tdx_cpu_cap_init_in_progress) + +/* For feature bit that KVM advertised through kvm_cpu_caps[]. */
I would say it
For feature bit that needs to be cap'ed by kvm_cpu_caps[]
arch/x86/kvm/vmx/tdx.c line 73
Quoted 11 lines
arch/x86/kvm/vmx/tdx.c +11
-0+#define TDX_CFG_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_caps |= feature_bit(name); \ +}) + +/* +* For feature bit KVM allows for TDX guests even though it is not advertised +* through kvm_cpu_caps[], e.g. MWAIT. +*/ +#define TDX_CFG_EXTRA_F (name) \
EXTRA doesn't sound like a fit name, though
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+({ \ +TDX_VALIDATE_CPU_CAP_USAGE (name); \ + tdx_cfg_extra_caps |=feature_bit (name); \ +}) + +#define tdx_cpu_cfg_cap_init(leaf, feature_initializers...) \ +do { \ + const u32 __maybe_unused tdx_cpu_cap_init_in_progress = leaf; \
arch/x86/kvm/vmx/tdx.c line 91
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+u32 tdx_cfg_extra_caps =0 ; \ +u32 tdx_cfg_caps =0 ; \ + \ +feature_initializers \ + tdx_cpu_cfg_caps[leaf] = (tdx_cfg_caps & kvm_cpu_caps[leaf]) | \ + tdx_cfg_extra_caps; \ +}while (0 ) + +/* +* Track only CPUID feature bits that are directly configurable by userspace.
the "by userspace" is misleading. It's just the directly configurable CPUID bits reported by TDX module.
arch/x86/kvm/vmx/tdx.c line 94
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +3
-0+* Features controlled by XFAM or ATTRIBUTES are excluded; userspace cannot +* enable them until KVM adds support for the corresponding control. +*/
I don't like the comments. How about somthing
/*
* Intialize tdx_cpu_cfg_caps[], which is list of CPUID features that
* KVM supports for TDX. It only covers the directly configurable CPIUD
* bits reported by TDX module. Features controlled by XFAM and
* ATTRIBUTES are maintained separately.
*/
arch/x86/kvm/vmx/tdx.c line 102
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+static void __init tdx_initialize_cpu_cfg_caps (void ) +{ +tdx_cpu_cfg_cap_init (CPUID_1_ECX , +TDX_CFG_EXTRA_F (MWAIT ), +TDX_CFG_F (TSC_DEADLINE_TIMER ), +TDX_CFG_F (AVX ), + TDX_CFG_F(F16C), + );
TDX 1.5.24 on SPR report configurable bits of CPUID_1_ECX as 0x31044988, which have
- bit 3 MWAIT
- bit 7 EST
- bit 8 TM2
- bit 11 SDBG
- bit 14 XTPR
- bit 18 DCA
- bit 24 TSC_DEADLINE_TIMER
- bit 28 AVX
- bit 29 F16C
but EST/TM2/SDBG/XTPR/DCA are not list here. I guess the reason is kvm_cpu_cap[] doesn't support it. If so, it seems to guard twice:
1. mentally/manually check if it a feature is supported in kvm_cpu_caps[]
2. kvm_cpu_caps guarding in tdx_cpu_cfg_cap_init().
I think 1) is not necessary, we can rely on 2)
BTW, this seems also breaks the current userspace after this series.
- Before, EST/TM2/SDBG/XTPR/DCA are allowed to be exposed to TD
- After, they are not.
If we cares CORE_CAPABILITIES in patch 2, why EST/TM2/SDBG/XTPR/DCA don't matter?
(I don't check the following leafs..)
Quoted 106 lines
arch/x86/kvm/vmx/tdx.c +96
-0+tdx_cpu_cfg_cap_init (CPUID_1_EDX , +TDX_CFG_F (MCE ), +TDX_CFG_F (MTRR ), +TDX_CFG_F (MCA ), + TDX_CFG_F(SELFSNOOP), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_0_EBX , +TDX_CFG_F (BMI1 ), +/* HLE */ +TDX_CFG_F (BMI2 ), +TDX_CFG_F (ERMS ), +/* RTM */ +TDX_CFG_F (AVX512F ), +TDX_CFG_F (AVX512DQ ), +TDX_CFG_F (ADX ), +TDX_CFG_F (AVX512IFMA ), +TDX_CFG_F (AVX512PF ), +TDX_CFG_F (AVX512ER ), +TDX_CFG_F (AVX512CD ), +TDX_CFG_F (AVX512BW ), + TDX_CFG_F(AVX512VL), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_ECX , +TDX_CFG_F (UMIP ), +/* WAITPKG */ +TDX_CFG_F (AVX512_VBMI2 ), +TDX_CFG_F (GFNI ), +TDX_CFG_F (VAES ), +TDX_CFG_F (VPCLMULQDQ ), +TDX_CFG_F (AVX512_VNNI ), +TDX_CFG_F (AVX512_BITALG ), +TDX_CFG_F (AVX512_VPOPCNTDQ ), +TDX_CFG_F (LA57 ), +TDX_CFG_F (RDPID ), + TDX_CFG_F(CLDEMOTE), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_EDX , +TDX_CFG_F (AVX512_4VNNIW ), +TDX_CFG_F (AVX512_4FMAPS ), +TDX_CFG_F (FSRM ), +TDX_CFG_F (AVX512_VP2INTERSECT ), +TDX_CFG_F (SERIALIZE ), + TDX_CFG_F(TSXLDTRK), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_1_EAX , +TDX_CFG_F (SHA512 ), +TDX_CFG_F (SM3 ), +TDX_CFG_F (SM4 ), +TDX_CFG_F (AVX_VNNI ), +TDX_CFG_F (AVX512_BF16 ), +TDX_CFG_F (CMPCCXADD ), +TDX_CFG_F (FZRM ), +TDX_CFG_F (FSRS ), +TDX_CFG_F (FSRC ), +TDX_CFG_F (LKGS ), +TDX_CFG_F (WRMSRNS ), +TDX_CFG_F (AMX_FP16 ), +TDX_CFG_F (AVX_IFMA ), +TDX_CFG_F (LAM ), + TDX_CFG_F(MOVRS), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_1_EDX , +TDX_CFG_F (AVX_VNNI_INT8 ), +TDX_CFG_F (AVX_NE_CONVERT ), +TDX_CFG_F (AMX_COMPLEX ), +TDX_CFG_F (AVX_VNNI_INT16 ), +TDX_CFG_F (PREFETCHITI ), + TDX_CFG_F(AVX10), + ); + +tdx_cpu_cfg_cap_init (CPUID_7_2_EDX , +TDX_CFG_F (DDPD_U ), + TDX_CFG_F(MCDT_NO), + ); + +tdx_cpu_cfg_cap_init (CPUID_1E_1_EAX , +TDX_CFG_F (AMX_FP8 ), +TDX_CFG_F (AMX_TF32 ), +TDX_CFG_F (AMX_AVX512 ), + TDX_CFG_F(AMX_MOVRS), + ); + +tdx_cpu_cfg_cap_init (CPUID_8000_0008_EBX , + TDX_CFG_F(WBNOINVD), + ); +} + +#undef TDX_CFG_F +#undef TDX_CFG_EXTRA_Fbool enable_tdx__ro_after_init ;module_param_named (tdx, enable_tdx, bool,0444 );@@ -3481,6 +3624,8 @@ int __init tdx_hardware_setup(void)return r; } +tdx_initialize_cpu_cfg_caps (); +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_tdx); vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size, sizeof(struct kvm_tdx));
Edgecombe, Rick P
maintainerQuoted 22 lines · Xiaoyao Li
Quoted 3 lines
Quoted 3 lines
Ideally the TDX save/restore would share code with normal VMs. On the other hand if we don't share enter/exit paths sufficiently, we may need to duplicate some save/restore in tdx code.
(Copy the FRED example here for reference)
>>>
>>> FRED is a useful concrete example. Under VMX, the FRED host state in >>>IA32_FRED_CONFIG,IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and >>> IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX module
>>> is expected to restore these MSRs on TDH.VP.ENTER return.IA32_FRED_RSP0
>>> andIA32_PL0_SSP(a.k.a.IA32_FRED_SSP0) are handled by software, so the
>>> TDX module is expected to clobber them on TDH.VP.ENTER return.What I get, is not matching VMX behavior but matching the behavior KVM will perform for VMX. They are based on the assumption that KVM will always enable the save/restore VMCS fields for a new feature. But I don't think we can guarantee it.
Quoted 7 lines
To me, "have TDX simply match VMX behavior" means:
1. if the VMX unconditionally save/restore a state, then TDX will do so.
2. if there are vm-entry/vm-exit load/save VMCS fields for a state, then provide the equivalent per-TD configurable interfaces which matches the VMCS fields.
What do you mean by this? Expose a TDX module interface to configure the clobber behavior for each feature with load/save configuration? That was similar to what we originally discussed, before pivoting to this solution.
I was thinking if you configured a feature (for example shadow stack), it would automatically set the VMCS save/restore settings associated with that feature. (VM_EXIT_LOAD_CET_STATE/VM_ENTRY_LOAD_CET_STATE)
This won't necessarily match KVM's behavior, because it could decide to not use the features. But we can probably get close with a simple rule that can make sense for all the VMMs.
If later we want a host clobber interface on top of the bit filtering, in order to minimize TDX special handling, we can probably add it later for features we care about. I'd think we don't need it right now.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Xiaoyao LiQuoted 21 lines · Xiaoyao Li
Quoted 19 lines · Binbin Wu
Add tdx_cpu_cfg_caps[] to track the subset of TDX directly configurable CPUID feature bits that KVM supports, and build the masks during TDX hardware setup via
tdx_initialize_cpu_cfg_caps().The TDX module reports the CPUID bits that the VMM can directly configure for a TD, but KVM cannot blindly expose all reported bits to userspace. Certain features imply additional architectural state, e.g. one or more MSRs, that KVM must explicitly manage across host/guest transitions to prevent host state corruption.
Today KVM relies on a hardcoded denylist, i.e. it clears a few known problematic bits, e.g. TSX and WAITPKG, and passes everything else through. A denylist is fundamentally fragile while an allowlist inverts the default, i.e. unknown configurable bits are hidden and not allowed to be enabled until KVM explicitly opts in.
Except for a few fixed-1 bits required for basic TDX support, host state clobbering features are either directly configurable or gated by TD ATTRIBUTES/XFAM.
Quoted 5 lines
Quoted 3 lines
Tracking only the directly configurable feature bits is therefore sufficient to serve the purpose while keeping the code footprint small.
I'm not clear how it is therefore sufficient. We at least need to explain that ATTRIBUTS/XFAM are validated separately by KVM already?
I was trying to say this by "gated by TD ATTRIBUTES/XFAM", I will describe it more clearly.
Quoted 47 lines
Quoted 32 lines
Organize tdx_cpu_cfg_caps[] following kvm_cpu_caps[] so that the masks can be built with the similar feature-name based initializers. CPUID registers that hold directly configurable non-feature (multi-bit) fields are handled separately.
The allowlist is consumed by later patches to filter
KVM_TDX_CAPABILITIESand to reject unsupported CPUID input toKVM_TDX_INIT_VM, so that newly introduced TDX directly configurable CPUID feature bits stay hidden from userspace until KVM explicitly opts in.Add comments as placeholders for HLE, RTM and WAITPKG, which KVM doesn't support for TDX yet.
Signed-off-by Binbin Wu <[email protected]>---
v3:
- Drop the new data structure in v2 and only track feature bits by following the organization of kvm_cpu_caps[], handle non-feature bits separately. (Sean)
- Use two versions of macros (TDX_CFG_F()VS.TDX_CFG_EXTRA_F()) to distinguish whether a supported TDX configurable CPUID bit should be checked against KVM's common cpu capabilities.
- AddAMX_COMPLEXsince it has been defined in the CPUID virtualization doc.
---
arch/x86/kvm/vmx/tdx.c| 145 +++++++++++++++++++++++++++++++++++++++++1 file changed, 145 insertions(+)arch/x86/kvm/vmx/tdx.c +0
-0arch/x86/kvm/vmx/tdx.c line 61
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-0@@ -52,6 +52,149 @@ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #a1 " 0x%llx, " #a2 ", 0x%llx, " #a3 " 0x%llx", \ a1, a2, a3) +static u32 tdx_cpu_cfg_caps[NR_KVM_CPU_CAPS] __ro_after_init; +static_assert (ARRAY_SIZE (tdx_cpu_cfg_caps) ==ARRAY_SIZE (kvm_cpu_caps)); + +#define TDX_VALIDATE_CPU_CAP_USAGE (name) \ + BUILD_BUG_ON(__feature_leaf(X86_FEATURE_##name) != \ + tdx_cpu_cap_init_in_progress) + +/* For feature bit that KVM advertised through kvm_cpu_caps[]. */ I would say it
For feature bit that needs to be cap'ed by kvm_cpu_caps[]
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c line 73
Quoted 11 lines
arch/x86/kvm/vmx/tdx.c +11
-0+#define TDX_CFG_F (name) \ +({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_caps |= feature_bit(name); \ +}) + +/* +* For feature bit KVM allows for TDX guests even though it is not advertised +* through kvm_cpu_caps[], e.g. MWAIT. +*/ +#define TDX_CFG_EXTRA_F (name) \EXTRA doesn't sound like a fit name, though
I also struggled with naming this macro and couldn't come up with a better one. Any suggestion?
Quoted 26 lines
Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +8
-0+({ \ + TDX_VALIDATE_CPU_CAP_USAGE(name); \ + tdx_cfg_extra_caps |= feature_bit(name); \ +}) + +#define tdx_cpu_cfg_cap_init(leaf, feature_initializers...) \ +do { \ + const u32 __maybe_unused tdx_cpu_cap_init_in_progress = leaf; \arch/x86/kvm/vmx/tdx.c line 91
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+ u32 tdx_cfg_extra_caps = 0; \ + u32 tdx_cfg_caps = 0; \ + \ + feature_initializers \ + tdx_cpu_cfg_caps[leaf] = (tdx_cfg_caps & kvm_cpu_caps[leaf]) | \ + tdx_cfg_extra_caps; \ +}while (0 ) + +/* +* Track only CPUID feature bits that are directly configurable by userspace. the "by userspace" is misleading. It's just the directly configurable CPUID bits reported by TDX module.
arch/x86/kvm/vmx/tdx.c line 94
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +3
-0+* Features controlled by XFAM or ATTRIBUTES are excluded; userspace cannot +* enable them until KVM adds support for the corresponding control. +*/
Quoted 9 lines
I don't like the comments. How about somthing
/*
* Intialize tdx_cpu_cfg_caps[], which is list of CPUID features that
* KVM supports for TDX. It only covers the directly configurable CPIUD
* bits reported by TDX module. Features controlled by XFAM and
* ATTRIBUTES are maintained separately.
*/
Thanks, it reads better.
Quoted 22 lines
arch/x86/kvm/vmx/tdx.c line 102
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+static void __init tdx_initialize_cpu_cfg_caps (void ) +{ + tdx_cpu_cfg_cap_init(CPUID_1_ECX, + TDX_CFG_EXTRA_F(MWAIT), + TDX_CFG_F(TSC_DEADLINE_TIMER), + TDX_CFG_F(AVX), + TDX_CFG_F(F16C), + );TDX 1.5.24 on SPR report configurable bits of
CPUID_1_ECXas0x31044988, which have- bit 3 MWAIT
- bit 7 EST
- bit 8 TM2
- bit 11 SDBG
- bit 14 XTPR
- bit 18 DCA
- bit 24TSC_DEADLINE_TIMER
- bit 28 AVX
- bit 29 F16C
Quoted 6 lines
but EST/TM2/SDBG/XTPR/DCA are not list here. I guess the reason is kvm_cpu_cap[] doesn't support it. If so, it seems to guard twice:
1. mentally/manually check if it a feature is supported in kvm_cpu_caps[]2.
kvm_cpu_capsguarding intdx_cpu_cfg_cap_init().I think 1) is not necessary, we can rely on 2)
In general, if a feature is not supported by the common KVM CPU caps, I prefer not to add it to the list to save a few lines of code, which probably is dead code, unless people find it too confusing.
I can add a comment to clarify this.
Quoted 4 lines
BTW, this seems also breaks the current userspace after this series.
- Before, EST/TM2/SDBG/XTPR/DCA are allowed to be exposed to TD
- After, they are not.
This does change the values returned by KVM_TDX_CAPABILITIES. However, my understanding is that userspace is generally expected to only configure features supported by both KVM and TDX, i.e. except for the features initialized via TDX_CFG_EXTRA_F(), userspace is not expected to configure features not advertised by kvm_cpu_caps[].
I can call this out in the changelog, and maybe also the doc for KVM_TDX_CAPABILITIES.
Quoted 2 lines
If we cares
CORE_CAPABILITIESin patch 2, why EST/TM2/SDBG/XTPR/DCA don't matter?
Because CORE_CAPABILITIES was previously defined as fixed-1 in some old spec and the QEMU marks it as fixed1. EST/TM2/SDBG/XTPR/DCA are not the case.
Quoted 27 lines · Rick P
Quoted 26 lines · Xiaoyao Li
Quoted 3 lines
Quoted 3 lines
Ideally the TDX save/restore would share code with normal VMs. On the other hand if we don't share enter/exit paths sufficiently, we may need to duplicate some save/restore in tdx code.
(Copy the FRED example here for reference)
>>>
>>> FRED is a useful concrete example. Under VMX, the FRED host state in >>>IA32_FRED_CONFIG,IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and >>> IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX
module
>>> is expected to restore these MSRs on TDH.VP.ENTER return.IA32_FRED_RSP0
>>> andIA32_PL0_SSP(a.k.a.IA32_FRED_SSP0) are handled by software,
so the
>>> TDX module is expected to clobber them on TDH.VP.ENTER return.What I get, is not matching VMX behavior but matching the behavior KVM will perform for VMX. They are based on the assumption that KVM will always enable the save/restore VMCS fields for a new feature. But I don't think we can guarantee it.
To me, "have TDX simply match VMX behavior" means:
1. if the VMX unconditionally save/restore a state, then TDX will do so.
Quoted 7 lines
Quoted 3 lines
2. if there are vm-entry/vm-exit load/save VMCS fields for a state, then provide the equivalent per-TD configurable interfaces which matches the VMCS fields.
What do you mean by this? Expose a TDX module interface to configure the clobber behavior for each feature with load/save configuration? That was similar to what we originally discussed, before pivoting to this solution.
yeah. This is what I meant. I was trying to show my literal understanding on "The proposal is to have TDX simply match VMX behavior". i.e., I don't think "have TDX simply match VMX behavior" is a good name/summary for what Binbin has proposed.
Quoted 7 lines
I was thinking if you configured a feature (for example shadow stack), it would automatically set the VMCS save/restore settings associated with that feature. (VM_EXIT_LOAD_CET_STATE/VM_ENTRY_LOAD_CET_STATE)
This won't necessarily match KVM's behavior, because it could decide to not use the features. But we can probably get close with a simple rule that can make sense for all the VMMs.
So the proposal is making TDX behave as if the relevant VMCS save/load controls (if any) are set around TDH.VP.ENTER.
In fact, what matters for host vmm is just the VM_EXIT_LOAD_XXX control. So the proposal becomes "If there is VM_EXIT_LOAD_XXX control for a state, TDX needs to restore the host state after TDH.VP.ENTER. If no such contorl, TDX sets the state to INIT state after TDH.VP.ENTER".
It's a fancy idea. And it provides a clear rule of how TDX handles states of a feature so that host VMM developers don't need to read the TDX module API to figure out what's the value of a state after TDH.VP.ENTER.
It's helpful for host VMM developers, though I'm not sure on TDX module developers.
Quoted 3 lines
If later we want a host clobber interface on top of the bit filtering, in order to minimize TDX special handling, we can probably add it later for features we care about. I'd think we don't need it right now.
Edgecombe, Rick P
maintainerQuoted 8 lines · Xiaoyao Li
So the proposal is making TDX behave as if the relevant VMCS save/load controls (if any) are set around TDH.VP.ENTER.
In fact, what matters for host vmm is just the
VM_EXIT_LOAD_XXXcontrol. So the proposal becomes "If there isVM_EXIT_LOAD_XXXcontrol for a state, TDX needs to restore the host state after TDH.VP.ENTER. If no such contorl, TDX sets the state to INIT state after TDH.VP.ENTER".
Quoted 6 lines
It's a fancy idea. And it provides a clear rule of how TDX handles states of a feature so that host VMM developers don't need to read the TDX module API to figure out what's the value of a state after TDH.VP.ENTER.
It's helpful for host VMM developers, though I'm not sure on TDX module developers.
I don't think they have agreed to it yet (Binbin?), but I think it mostly works this way already. Why do you think it is a burden on TDX module developers?
Also, I think this can be the guideline. We could have exceptions.
Quoted 13 lines · Rick P
Quoted 12 lines · Xiaoyao Li
So the proposal is making TDX behave as if the relevant VMCS save/load controls (if any) are set around TDH.VP.ENTER.
In fact, what matters for host vmm is just the
VM_EXIT_LOAD_XXXcontrol. So the proposal becomes "If there isVM_EXIT_LOAD_XXXcontrol for a state, TDX needs to restore the host state after TDH.VP.ENTER. If no such contorl, TDX sets the state to INIT state after TDH.VP.ENTER".It's a fancy idea. And it provides a clear rule of how TDX handles states of a feature so that host VMM developers don't need to read the TDX module API to figure out what's the value of a state after TDH.VP.ENTER.
Quoted 5 lines
Quoted 2 lines
It's helpful for host VMM developers, though I'm not sure on TDX module developers.
I don't think they have agreed to it yet (Binbin?), but I think it mostly works this way already. Why do you think it is a burden on TDX module developers?
Because it can bring confusion to TDX module developers. VM_EXIT_LOAD_XXX control is used by SEAM VMCS to load states for TDX module execution context. And I think for most features SEAM VMCS doesn't set it (I don't check it though).
SEAMCALL is kind of a VM exit, and SEAMRET is kind of a VM entry. To automatically save and restore the host state, what TDX needs are VM_EXIT_SAVE_XXX, VM_ENTRY_LOAD_XXX, and vmcs guest state for XXX.
Edgecombe, Rick P
maintainerQuoted 4 lines · Xiaoyao Li
Quoted 3 lines
I don't think they have agreed to it yet (Binbin?), but I think it mostly works this way already. Why do you think it is a burden on TDX module developers?
Quoted 8 lines
Because it can bring confusion to TDX module developers.
VM_EXIT_LOAD_XXXcontrol is used by SEAM VMCS to load states for TDX module execution context. And I think for most features SEAM VMCS doesn't set it (I don't check it though).SEAMCALL is kind of a VM exit, and SEAMRET is kind of a VM entry. To automatically save and restore the host state, what TDX needs are
VM_EXIT_SAVE_XXX,VM_ENTRY_LOAD_XXX, and vmcs guest state for XXX.
I'd think we could avoid adding options for configuration that won't be used. If we did have a clobber control interface, matching VMX bits is an interesting idea.
I'm a bit torn between wanting to fix the area once and for all with a full solution, and wanting to get this increasingly blocking CPUID bit fix in. I'm leaning towards just do the bit filtering and give the save/restore guidelines.
Then we can do a save/restore control later if we find the guidelines are not sufficient.
Quoted 18 lines · Rick P
Quoted 4 lines · Xiaoyao Li
Quoted 3 lines
I don't think they have agreed to it yet (Binbin?), but I think it mostly works this way already. Why do you think it is a burden on TDX module developers?
Quoted 8 lines
Because it can bring confusion to TDX module developers.
VM_EXIT_LOAD_XXXcontrol is used by SEAM VMCS to load states for TDX module execution context. And I think for most features SEAM VMCS doesn't set it (I don't check it though).SEAMCALL is kind of a VM exit, and SEAMRET is kind of a VM entry. To automatically save and restore the host state, what TDX needs are
VM_EXIT_SAVE_XXX,VM_ENTRY_LOAD_XXX, and vmcs guest state for XXX.I'd think we could avoid adding options for configuration that won't be used. If we did have a clobber control interface, matching VMX bits is an interesting idea.
Quoted 5 lines
I'm a bit torn between wanting to fix the area once and for all with a full solution, and wanting to get this increasingly blocking CPUID bit fix in.
I'm
leaning towards just do the bit filtering and give the save/restore guidelines.
I agree.
I always believe the bit filtering introduced by this series makes KVM safer and it's anyway useful, while how TDX/TDX module save/restore host states can be another separate topic.
Quoted 2 lines
Then we can do a save/restore control later if we find the guidelines are not sufficient.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Binbin WuQuoted 21 lines · Binbin Wu
Quoted 21 lines
arch/x86/kvm/vmx/tdx.c line 102
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+static void __init tdx_initialize_cpu_cfg_caps (void ) +{ + tdx_cpu_cfg_cap_init(CPUID_1_ECX, + TDX_CFG_EXTRA_F(MWAIT), + TDX_CFG_F(TSC_DEADLINE_TIMER), + TDX_CFG_F(AVX), + TDX_CFG_F(F16C), + );TDX 1.5.24 on SPR report configurable bits of
CPUID_1_ECXas0x31044988, which have- bit 3 MWAIT
- bit 7 EST
- bit 8 TM2
- bit 11 SDBG
- bit 14 XTPR
- bit 18 DCA
- bit 24TSC_DEADLINE_TIMER
- bit 28 AVX
- bit 29 F16C
Quoted 7 lines
Quoted 6 lines
but EST/TM2/SDBG/XTPR/DCA are not list here. I guess the reason is kvm_cpu_cap[] doesn't support it. If so, it seems to guard twice:
1. mentally/manually check if it a feature is supported in kvm_cpu_caps[]2.
kvm_cpu_capsguarding intdx_cpu_cfg_cap_init().I think 1) is not necessary, we can rely on 2)
In general, if a feature is not supported by the common KVM CPU caps,
For kvm-intel.ko, kvm_cpu_caps[] just means the supported CPUID features for VMX VMs. Treat it as the common KVM CPU caps is a bit arguable.
Quoted 2 lines
I prefer not
to add it to the list to save a few lines of code, which probably is dead code,
I don't think it's dead code. It shows that these features are virtualizable to TDs from the POV. of TDX.
In the end, they might be disallowed to be configured to TDs because KVM doesn't allow them for VMX VMs. This is also the point I want to discuss. Do we really want to make such restriction that KVM cannot enable/allow a feature for TDs unless KVM first enables/allows it for VMX VMs? What's reason behind it?
Quoted 6 lines
unless people find it too confusing.
I can add a comment to clarify this.Quoted 3 lines
BTW, this seems also breaks the current userspace after this series.
- Before, EST/TM2/SDBG/XTPR/DCA are allowed to be exposed to TD
- After, they are not.
Quoted 6 lines
This does change the values returned by
KVM_TDX_CAPABILITIES. However, my understanding is that userspace is generally expected to only configure features supported by both KVM and TDX, i.e. except for the features initialized viaTDX_CFG_EXTRA_F(), userspace is not expected to configure features not advertised by kvm_cpu_caps[].
I can call this out in the changelog, and maybe also the doc forKVM_TDX_CAPABILITIES.
yeah. This changes KVM's behavior and we definitely need to call it out, and provide justification.
Quoted 3 lines
Quoted 1 line
If we cares
CORE_CAPABILITIESin patch 2, why EST/TM2/SDBG/XTPR/DCA don't matter?Because
CORE_CAPABILITIESwas previously defined as fixed-1 in some old spec and the QEMU marks it as fixed1. EST/TM2/SDBG/XTPR/DCA are not the case.
I see. You added patch 2 because without it QEMU breaks. While for EST/TM2/SDBG/XTPR/DCA, QEMU doesn't break after they are turned to non-configurable. QEMU cannot represent all the userspace VMM. It still has the potential to breaks other userspace VMMs.
Edgecombe, Rick P
maintainerQuoted 5 lines · Xiaoyao Li
Quoted 4 lines · Binbin Wu
Quoted 4 lines
Quoted 4 lines
So if the approach in this series is taken, I think we still need such an opt- in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Yep. Can we think about what it would look like? Easiest would be a bit passed in
TDH_SYS_CONFIG. But thenarch/x86is saying how KVM will behave. Ok to me, for the simplicity. Could come with a nice comment.Can you just treat current KVM behavior of allowing userspace to enable any configurable bits as the bug of KVM and backport this series as Binbin suggested below? Instead of introducing more opt-in knobs.
Ok, so if we are agreed on the other branch of the thread, the only big question
is: Do we want an opt-in for future clobbering CPUID bits.
I think either is ok. I don't love the precedent that we asserted that no new clobber bits could be added without opt-in, and then we would backport changes to allow this anyway. But on pure code, the backport would be simpler in the long term. If you guys are strongly in favor, I can agree.
Are we sure no other VMM needs an opt-in, before finalizing it though? Binbin, can flag this to the TDX module team?
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Xiaoyao LiQuoted 30 lines · Xiaoyao Li
Quoted 21 lines · Binbin Wu
Quoted 21 lines
arch/x86/kvm/vmx/tdx.c line 102
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+static void __init tdx_initialize_cpu_cfg_caps (void ) +{ + tdx_cpu_cfg_cap_init(CPUID_1_ECX, + TDX_CFG_EXTRA_F(MWAIT), + TDX_CFG_F(TSC_DEADLINE_TIMER), + TDX_CFG_F(AVX), + TDX_CFG_F(F16C), + );TDX 1.5.24 on SPR report configurable bits of
CPUID_1_ECXas0x31044988, which have- bit 3 MWAIT
- bit 7 EST
- bit 8 TM2
- bit 11 SDBG
- bit 14 XTPR
- bit 18 DCA
- bit 24TSC_DEADLINE_TIMER
- bit 28 AVX
- bit 29 F16CQuoted 7 lines
Quoted 6 lines
but EST/TM2/SDBG/XTPR/DCA are not list here. I guess the reason is kvm_cpu_cap[] doesn't support it. If so, it seems to guard twice:
1. mentally/manually check if it a feature is supported in kvm_cpu_caps[]2.
kvm_cpu_capsguarding intdx_cpu_cfg_cap_init().I think 1) is not necessary, we can rely on 2)
In general, if a feature is not supported by the common KVM CPU caps,
Quoted 8 lines
For kvm-intel.ko, kvm_cpu_caps[] just means the supported CPUID features for VMX VMs. Treat it as the common KVM CPU caps is a bit arguable.
Quoted 2 lines
I prefer not
to add it to the list to save a few lines of code, which probably is dead code,I don't think it's dead code. It shows that these features are virtualizable to TDs from the POV. of TDX.
It depends on whether KVM allows userspace to set features for TDs that are not support for non-TDX VMs (,except for a few exceptions).
In this version, TDX_CFG_F() already check against kvm_cpu_caps[], if these features are not in kvm_cpu_caps[], it will not be exposed to userspace anyway.
Quoted 6 lines
In the end, they might be disallowed to be configured to TDs because KVM doesn't allow them for VMX VMs. This is also the point I want to discuss. Do we really want to make such restriction that KVM cannot enable/allow a feature for TDs unless KVM first enables/allows it for VMX VMs? What's reason behind it?
Sean mentioned it that "generally speaking, KVM shouldn't allow features that KVM doesn't support for non-TDX VMs" in
https://lore.kernel.org/kvm/[email protected]/
Quoted 17 lines
Quoted 7 lines
unless people find it too confusing.
I can add a comment to clarify this.Quoted 3 lines
BTW, this seems also breaks the current userspace after this series.
- Before, EST/TM2/SDBG/XTPR/DCA are allowed to be exposed to TD
- After, they are not.Quoted 6 lines
This does change the values returned by
KVM_TDX_CAPABILITIES. However, my understanding is that userspace is generally expected to only configure features supported by both KVM and TDX, i.e. except for the features initialized viaTDX_CFG_EXTRA_F(), userspace is not expected to configure features not advertised by kvm_cpu_caps[].
I can call this out in the changelog, and maybe also the doc forKVM_TDX_CAPABILITIES.yeah. This changes KVM's behavior and we definitely need to call it out, and provide justification.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
If we cares
CORE_CAPABILITIESin patch 2, why EST/TM2/SDBG/XTPR/DCA don't matter?Because
CORE_CAPABILITIESwas previously defined as fixed-1 in some old spec and the QEMU marks it as fixed1. EST/TM2/SDBG/XTPR/DCA are not the case.I see. You added patch 2 because without it QEMU breaks. While for EST/TM2/SDBG/XTPR/DCA, QEMU doesn't break after they are turned to non-configurable. QEMU cannot represent all the userspace VMM. It still has the potential to breaks other userspace VMMs.
I think the risk is pretty low.
I am not sure if Sean could provide some insight about this in google's userspace VMM.
Quoted 17 lines · Rick P
Quoted 5 lines · Xiaoyao Li
Quoted 4 lines · Binbin Wu
Quoted 4 lines
Quoted 4 lines
So if the approach in this series is taken, I think we still need such an opt- in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Yep. Can we think about what it would look like? Easiest would be a bit passed in
TDH_SYS_CONFIG. But thenarch/x86is saying how KVM will behave. Ok to me, for the simplicity. Could come with a nice comment.Can you just treat current KVM behavior of allowing userspace to enable any configurable bits as the bug of KVM and backport this series as Binbin suggested below? Instead of introducing more opt-in knobs.
Ok, so if we are agreed on the other branch of the thread, the only big question
is: Do we want an opt-in for future clobbering CPUID bits.
Quoted 7 lines
I think either is ok. I don't love the precedent that we asserted that no new clobber bits could be added without opt-in, and then we would backport changes to allow this anyway. But on pure code, the backport would be simpler in the long term. If you guys are strongly in favor, I can agree.
Are we sure no other VMM needs an opt-in, before finalizing it though? Binbin, can flag this to the TDX module team?
AFAIK, no other VMM complained about the host clobbering issue. I will check it with the TDX module team.
Edgecombe, Rick P
maintainerRe: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Binbin WuQuoted 8 lines · Binbin Wu
Quoted 4 lines
I see. You added patch 2 because without it QEMU breaks. While for EST/TM2/SDBG/XTPR/DCA, QEMU doesn't break after they are turned to non-configurable. QEMU cannot represent all the userspace VMM. It still has the potential to breaks other userspace VMMs.
I think the risk is pretty low.
I am not sure if Sean could provide some insight about this in google's userspace VMM.
Can't we fix the issue if we are wrong?
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
Quoted 11 lines · Rick P
Quoted 8 lines · Binbin Wu
Quoted 4 lines
I see. You added patch 2 because without it QEMU breaks. While for EST/TM2/SDBG/XTPR/DCA, QEMU doesn't break after they are turned to non-configurable. QEMU cannot represent all the userspace VMM. It still has the potential to breaks other userspace VMMs.
I think the risk is pretty low.
I am not sure if Sean could provide some insight about this in google's userspace VMM.Can't we fix the issue if we are wrong?
I think it could be fixed to add the missing bits (if any) as bug fix.
Quoted 9 lines · Rick P
Quoted 8 lines · Xiaoyao Li
So the proposal is making TDX behave as if the relevant VMCS save/load controls (if any) are set around TDH.VP.ENTER.
In fact, what matters for host vmm is just the
VM_EXIT_LOAD_XXXcontrol. So the proposal becomes "If there isVM_EXIT_LOAD_XXXcontrol for a state, TDX needs to restore the host state after TDH.VP.ENTER. If no such contorl, TDX sets the state to INIT state after TDH.VP.ENTER".
Quoted 8 lines
Quoted 6 lines
It's a fancy idea. And it provides a clear rule of how TDX handles states of a feature so that host VMM developers don't need to read the TDX module API to figure out what's the value of a state after TDH.VP.ENTER.
It's helpful for host VMM developers, though I'm not sure on TDX module developers.
I don't think they have agreed to it yet (Binbin?),
Not yet. We just brought the topic to the community for discussion first.
Quoted 4 lines
but I think it mostly works
this way already. Why do you think it is a burden on TDX module developers?Also, I think this can be the guideline. We could have exceptions.
Re: [PATCH v3 2/4] KVM: TDX: Report CORE_CAPABILITIES as configurable
In reply to Binbin Wuarch/x86/kvm/vmx/tdx.c static void __init tdx_initialize_cpu_cfg_caps(void) · line 152
Quoted 11 lines · Binbin Wu
Reporting
CORE_CAPABILITIESas configurable keeps userspace able to enable the bit across the fixed-1 => configurable transition, and lets userspace infer that the bit is no longer fixed-1 so it can adjust its expectations.
...arch/x86/kvm/vmx/tdx.c +6
-0@@ -147,6 +147,12 @@ static void __init tdx_initialize_cpu_cfg_caps(void) +/* +* KVM does not support MSR_IA32_CORE_CAPS, but older TDX specs +* define this bit as fixed-1. Report it as configurable so +* userspace can know the feature is no longer a fixed-1 bit. +*/ + TDX_CFG_EXTRA_F(CORE_CAPABILITIES),
Would a mask of the bits that are fixed-1 across the modules KVM supports today be worth carrying, or adding to the allowed set?
I assume it would be hardcoded, just as the allowed list is.
If a module update later made one of them configurable, userspace would keep the ability to set it -- something it has today with the denylist. And since such bits are already enabled in every TD running now, continuing to accept them can't turn on anything that isn't already on.
A bit that becomes fixed-1 in the future and configurable after a subsequent module update would still need a patch to the mask, just as this patch does. Is CORE_CAPABILITIES the only fixed-1 case so far?
Re: [PATCH v3 2/4] KVM: TDX: Report CORE_CAPABILITIES as configurable
Quoted 16 lines · Kishen Maloor
arch/x86/kvm/vmx/tdx.c static void __init tdx_initialize_cpu_cfg_caps(void) · line 152
Quoted 11 lines · Binbin Wu
Reporting
CORE_CAPABILITIESas configurable keeps userspace able to enable the bit across the fixed-1 => configurable transition, and lets userspace infer that the bit is no longer fixed-1 so it can adjust its expectations.
...arch/x86/kvm/vmx/tdx.c +6
-0@@ -147,6 +147,12 @@ static void __init tdx_initialize_cpu_cfg_caps(void) +/* +* KVM does not support MSR_IA32_CORE_CAPS, but older TDX specs +* define this bit as fixed-1. Report it as configurable so +* userspace can know the feature is no longer a fixed-1 bit. +*/ + TDX_CFG_EXTRA_F(CORE_CAPABILITIES),Would a mask of the bits that are fixed-1 across the modules KVM supports today be worth carrying, or adding to the allowed set?
I assume it would be hardcoded, just as the allowed list is.
This version doesn't include fixed-1 bits or other non-"directly configurable" bits to minimize the code footprint.
Quoted 5 lines
If a module update later made one of them configurable, userspace would keep the ability to set it -- something it has today with the denylist. And since such bits are already enabled in every TD running now, continuing to accept them can't turn on anything that isn't already on.
After discussions with the TDX module team, there is one potential backward compatibility issue regarding x86 feature deprecation. I raised the it as in the cover letter and there were discussions.
https://lore.kernel.org/kvm/[email protected]/
Quoted 4 lines
A bit that becomes fixed-1 in the future and configurable after a subsequent module update would still need a patch to the mask, just as this patch does. Is
CORE_CAPABILITIESthe only fixed-1 case so far?
Actually this is the #VE reduction on fixed-1 bit, which changes a previous fixed-1 bit to directly configurable.
There are several cases of this category, i.e. MCA/MCE/MTRR/CORE_CAPABILITIES. However, CORE_CAPABILITIES is the only case that KVM doesn't support for non-TDX VMs
I will add these info to the change log.
Re: [PATCH v3 2/4] KVM: TDX: Report CORE_CAPABILITIES as configurable
Quoted 10 lines · Binbin Wu
Quoted 1 line · Kishen Maloor
On 8/26/26 8:18 PM, Binbin Wu wrote:
...
Actually this is the #VE reduction on fixed-1 bit, which changes a previous fixed-1 bit to directly configurable.
There are several cases of this category, i.e. MCA/MCE/MTRR/CORE_CAPABILITIES. However,CORE_CAPABILITIESis the only case that KVM doesn't support for non-TDX VMs
I will add these info to the change log.
The cover letter highlights the deprecation case, but it sounds like you hit this now due to #VE reduction. Is further #VE reduction expected? If this is altogether rare, then yeah, added measures aren't necessary.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Binbin WuQuoted 35 lines · Binbin Wu
Quoted 34 lines · Xiaoyao Li
Quoted 21 lines · Binbin Wu
Quoted 21 lines
arch/x86/kvm/vmx/tdx.c line 102
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+static void __init tdx_initialize_cpu_cfg_caps (void ) +{ + tdx_cpu_cfg_cap_init(CPUID_1_ECX, + TDX_CFG_EXTRA_F(MWAIT), + TDX_CFG_F(TSC_DEADLINE_TIMER), + TDX_CFG_F(AVX), + TDX_CFG_F(F16C), + );TDX 1.5.24 on SPR report configurable bits of
CPUID_1_ECXas0x31044988, which have- bit 3 MWAIT
- bit 7 EST
- bit 8 TM2
- bit 11 SDBG
- bit 14 XTPR
- bit 18 DCA
- bit 24TSC_DEADLINE_TIMER
- bit 28 AVX
- bit 29 F16CQuoted 7 lines
Quoted 6 lines
but EST/TM2/SDBG/XTPR/DCA are not list here. I guess the reason is kvm_cpu_cap[] doesn't support it. If so, it seems to guard twice:
1. mentally/manually check if it a feature is supported in kvm_cpu_caps[]2.
kvm_cpu_capsguarding intdx_cpu_cfg_cap_init().I think 1) is not necessary, we can rely on 2)
In general, if a feature is not supported by the common KVM CPU caps,
For kvm-intel.ko, kvm_cpu_caps[] just means the supported CPUID features for VMX VMs. Treat it as the common KVM CPU caps is a bit arguable.
Quoted 2 lines
I prefer not
to add it to the list to save a few lines of code, which probably is dead code,
Quoted 8 lines
Quoted 2 lines
I don't think it's dead code. It shows that these features are virtualizable to TDs from the POV. of TDX.
It depends on whether KVM allows userspace to set features for TDs that are not support for non-TDX VMs (,except for a few exceptions).
In this version,
TDX_CFG_F()already check against kvm_cpu_caps[], if these features are not in kvm_cpu_caps[], it will not be exposed to userspace anyway.
I still think the reasoning that we omit them because they are not contained in kvm_cpu_caps[] sounds not right. Based on it, so when we are going to add a new feature for TDX, we need to first manually check the KVM code to see if that feature is contained in kvm_cpu_caps[] already. If not, we just don't add it to TDX's list. Then why need to cap the result kvm_cpu_caps[] for TDX_CFG_F() again? Just for safety in case human make mistake and misread the code of kvm_cpu_caps[]?
I think they are two independent steps:
1. list the CPUID features that KVM can support for TDs.
2. apply additional restrictions, e.g., if a feature is not allowed for non-TDX VMs, it cannot be allowed for TDs.
Quoted 8 lines
Quoted 5 lines
In the end, they might be disallowed to be configured to TDs because KVM doesn't allow them for VMX VMs. This is also the point I want to discuss. Do we really want to make such restriction that KVM cannot enable/allow a feature for TDs unless KVM first enables/allows it for VMX VMs? What's reason behind it?
Sean mentioned it that "generally speaking, KVM shouldn't allow features that KVM doesn't support for non-TDX VMs" in
https://lore.kernel.org/kvm/[email protected]/
For existing features, it might make some sense. But for new features, I don't think so. It defines the enabling order for new features that we must enable a feature for non-TDX VMs first and then TDs. And people might want to bypass this rule by abusing the TDX_CFG_EXTRA_F() when only one line of TDX_CFG_EXTRA_F() is enough to enable a feature for TDs but more effort required to enable it for non-TDX VMs.
Maybe I miss somthing. I would like to see stronger reasons for such decision.
Re: [PATCH v3 3/4] KVM: TDX: Filter configurable CPUID bits
In reply to Binbin WuQuoted 59 lines · Binbin Wu
Filter the directly configurable CPUID bits reported through
KVM_TDX_CAPABILITIESagainst KVM's TDX allowlist, and drop the hardcoded denylist based filtering.The TDX module reports all directly configurable CPUID bits that it supports for a TD, but KVM must not expose bits that it doesn't support, as blindly exposing a host state clobbering feature can lead to host state corruption. The existing denylist, which clears only TSX and WAITPKG, is not fail-safe.
Add
tdx_get_allowed_cfg_cpuid_mask()to get the mask of directly configurable bits allowed by KVM for a given CPUID register, covering both feature bits, which come from tdx_cpu_cfg_caps[], and non-feature bits, which are enumerated at runtime. Apply the mask to every CPUID register reported throughKVM_TDX_CAPABILITIES.With the allowlist in place, newly introduced TDX directly configurable CPUID bits stay hidden from userspace until KVM explicitly opts in.
Signed-off-by Binbin Wu <[email protected]>---
v3:
- Handle non-feature leafs at runtime. (Sean)
- Add CPUID.0x24.0.EBX[7:0] into allow list. There is a mismatch of the description about CPUID.0x24.0.EBX[7:0], which is listed as "XFAM &CPUID_Enabled& Native" but should be "XFAM &CPUID_Enabled& Configured & Native".--- arch/x86/kvm/vmx/tdx.c | 84 +++++++++++++++++++++++++++++++++--------- 1 file changed, 66 insertions(+), 18 deletions(-)arch/x86/kvm/vmx/tdx.c +3
-8@@ -277,34 +277,76 @@ static bool has_tsx(const struct kvm_cpuid_entry2 *entry) (entry->ebx &TDX_FEATURE_TSX ); } -static void clear_tsx (struct kvm_cpuid_entry2 *entry) -{ - entry->ebx &= ~TDX_FEATURE_TSX ; -} -static bool has_waitpkg (const struct kvm_cpuid_entry2 *entry) {return entry->function ==7 && entry->index ==0 && (entry->ecx &__feature_bit (X86_FEATURE_WAITPKG )); } -static void clear_waitpkg (struct kvm_cpuid_entry2 *entry) +static bool tdx_unsupported_cpuid (const struct kvm_cpuid_entry2 *entry) { - entry->ecx &= ~__feature_bit (X86_FEATURE_WAITPKG ); +return has_tsx (entry) ||has_waitpkg (entry); } -static void tdx_clear_unsupported_cpuid (struct kvm_cpuid_entry2 *entry) +#define TDX_CPUID_ALL_ALLOWED_MASK GENMASK_U32(31, 0)
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 299
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-2+ +static u32 tdx_cfg_non_feature_mask (u32 function,u32 index,int reg) { -if (has_tsx (entry)) -clear_tsx (entry); +/* +* For a leaf/subleaf/register that will never be repurposed to hold +* feature bits, it's safe to return TDX_CPUID_ALL_ALLOWED_MASK, i.e. +* leave the TDX module's CPUID config mask intact. +*/
I don't think blindly return TDX_CPUID_ALL_ALLOWED_MASK, i.e. all-1s, is a good idea. It's just like the current behavior that KVM doesn't gate anything and allows userspace to set anything that is allowed by TDX module. For example, ...
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 303
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4
-0+switch (function) { +case 1 : +if (reg ==CPUID_EAX || reg ==CPUID_EBX ) +return TDX_CPUID_ALL_ALLOWED_MASK ;
... TDX module returns 0x0fff3fff for CPUID.1.EAX currently. If KVM makes the mask as all-1s, then if in the future the reserved field [15:14] and [31:28] are defined for new things and new TDX module starts to report them as configurable, then the bits will be configurable by userspace on old kernels while we don't know if its safe for KVM/kernel.
Quoted 17 lines
arch/x86/kvm/vmx/tdx.c +17
-0+return 0 ; +case 4 : +case 0x18 : +case 0x1f : +return TDX_CPUID_ALL_ALLOWED_MASK ; +case 0x24 : +if (index ==0 && reg ==CPUID_EBX ) +return GENMASK_U32 (7 ,0 ); +return 0 ; +case 0x80000008 : +if (reg ==CPUID_EAX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +default : +return 0 ; + } +}
Re: [PATCH v3 2/4] KVM: TDX: Report CORE_CAPABILITIES as configurable
In reply to Kishen MaloorQuoted 13 lines · Kishen Maloor
Quoted 10 lines · Binbin Wu
Quoted 1 line · Kishen Maloor
On 8/26/26 8:18 PM, Binbin Wu wrote:
...
Actually this is the #VE reduction on fixed-1 bit, which changes a previous fixed-1 bit to directly configurable.
There are several cases of this category, i.e. MCA/MCE/MTRR/CORE_CAPABILITIES. However,CORE_CAPABILITIESis the only case that KVM doesn't support for non-TDX VMs
I will add these info to the change log.
Quoted 3 lines
The cover letter highlights the deprecation case, but it sounds like you hit this now due to #VE reduction. Is further #VE reduction expected? If this is altogether rare, then yeah, added measures aren't necessary.
No new #VE reductions are expected.
Re: [PATCH v3 3/4] KVM: TDX: Filter configurable CPUID bits
In reply to Xiaoyao LiQuoted 6 lines · Xiaoyao Li
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +3
-2+static u32 tdx_cfg_non_feature_mask (u32 function,u32 index,int reg) { -if (has_tsx (entry)) -clear_tsx (entry); +/* +* For a leaf/subleaf/register that will never be repurposed to hold
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 299
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +3
-0+* feature bits, it's safe to return TDX_CPUID_ALL_ALLOWED_MASK, i.e. +* leave the TDX module's CPUID config mask intact. +*/ I don't think blindly return
TDX_CPUID_ALL_ALLOWED_MASK, i.e. all-1s, is a good idea. It's just like the current behavior that KVM doesn't gate anything and allows userspace to set anything that is allowed by TDX module. For example, ...
Sean suggested that "Realistically, CPUID.0x1.E{A,B}X are never going to be repurposed to hold feature bits, and so generating a mask of allowed bits adds unnecessary cognitive load and maintenance. Ditto for CPUID 0x4, 0x18, and 0x1F."
https://lore.kernel.org/kvm/[email protected]/
And I added 0x80000008.EAX as well.
Quoted 30 lines
arch/x86/kvm/vmx/tdx.c static bool has_tsx(const struct kvm_cpuid_entry2 *entry) · line 303
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +4
-0+switch (function) { +case 1 : +if (reg ==CPUID_EAX || reg ==CPUID_EBX ) +return TDX_CPUID_ALL_ALLOWED_MASK ;... TDX module returns
0x0fff3ffffor CPUID.1.EAX currently. If KVM makes the mask as all-1s, then if in the future the reserved field [15:14] and [31:28] are defined for new things and new TDX module starts to report them as configurable, then the bits will be configurable by userspace on old kernels while we don't know if its safe for KVM/kernel.Quoted 17 lines
arch/x86/kvm/vmx/tdx.c +17
-0+return 0 ; +case 4 : +case 0x18 : +case 0x1f : +return TDX_CPUID_ALL_ALLOWED_MASK ; +case 0x24 : +if (index ==0 && reg ==CPUID_EBX ) +return GENMASK_U32 (7 ,0 ); +return 0 ; +case 0x80000008 : +if (reg ==CPUID_EAX ) +return TDX_CPUID_ALL_ALLOWED_MASK ; +return 0 ; +default : +return 0 ; + } +}
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Xiaoyao LiQuoted 41 lines · Xiaoyao Li
Quoted 40 lines · Binbin Wu
Quoted 28 lines · Xiaoyao Li
Quoted 27 lines · Binbin Wu
Quoted 27 lines
arch/x86/kvm/vmx/tdx.c line 102
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +8
-0+static void __init tdx_initialize_cpu_cfg_caps (void ) +{ + tdx_cpu_cfg_cap_init(CPUID_1_ECX, + TDX_CFG_EXTRA_F(MWAIT), + TDX_CFG_F(TSC_DEADLINE_TIMER), + TDX_CFG_F(AVX), + TDX_CFG_F(F16C), + );TDX 1.5.24 on SPR report configurable bits of
CPUID_1_ECXas0x31044988, which have- bit 3 MWAIT
- bit 7 EST
- bit 8 TM2
- bit 11 SDBG
- bit 14 XTPR
- bit 18 DCA
- bit 24TSC_DEADLINE_TIMER
- bit 28 AVX
- bit 29 F16Cbut EST/TM2/SDBG/XTPR/DCA are not list here. I guess the reason is kvm_cpu_cap[] doesn't support it. If so, it seems to guard twice:
1. mentally/manually check if it a feature is supported in kvm_cpu_caps[]2.
kvm_cpu_capsguarding intdx_cpu_cfg_cap_init().I think 1) is not necessary, we can rely on 2)
Quoted 8 lines
Quoted 1 line
In general, if a feature is not supported by the common KVM CPU caps,
For kvm-intel.ko, kvm_cpu_caps[] just means the supported CPUID features for VMX VMs. Treat it as the common KVM CPU caps is a bit arguable.
Quoted 2 lines
I prefer not
to add it to the list to save a few lines of code, which probably is dead code,I don't think it's dead code. It shows that these features are virtualizable to TDs from the POV. of TDX.
It depends on whether KVM allows userspace to set features for TDs that are not support for non-TDX VMs (,except for a few exceptions).
Quoted 6 lines
Quoted 3 lines
In this version,
TDX_CFG_F()already check against kvm_cpu_caps[], if these features are not in kvm_cpu_caps[], it will not be exposed to userspace anyway.I still think the reasoning that we omit them because they are not contained in kvm_cpu_caps[] sounds not right.
TBH,I omitted these bits for simplicity.
It's not just adding a TDX_CFG_F(XXX) for a feature, but also need to add new X86_FEATURE_XXX definitions sometimes.
E.g. CPUID.7.0.ECX[0] (PREFETCHWT1), which is directly configurable and not supported by KVM. It has no definition in <asm/cpufeatures.h>.
I am not sure that's worth it.
Quoted 5 lines
Based on it, so when we are
going to add a new feature for TDX, we need to first manually check the KVM code to see if that feature is contained in kvm_cpu_caps[] already. If not, we just don't add it to TDX's list. Then why need to cap the result kvm_cpu_caps[] forTDX_CFG_F()again?
This is due to "generally speaking, KVM shouldn't allow features that KVM doesn't support for non-TDX VMs".
I'm hoping Sean can weigh in on whether KVM should restrict TDX's CPU capabilities in this way.
Quoted 17 lines
Just for safety in case human make
mistake and misread the code of kvm_cpu_caps[]?I think they are two independent steps:
1. list the CPUID features that KVM can support for TDs.
2. apply additional restrictions, e.g., if a feature is not allowed for non-TDX VMs, it cannot be allowed for TDs.Quoted 8 lines
Quoted 5 lines
In the end, they might be disallowed to be configured to TDs because KVM doesn't allow them for VMX VMs. This is also the point I want to discuss. Do we really want to make such restriction that KVM cannot enable/allow a feature for TDs unless KVM first enables/allows it for VMX VMs? What's reason behind it?
Sean mentioned it that "generally speaking, KVM shouldn't allow features that KVM doesn't support for non-TDX VMs" in
https://lore.kernel.org/kvm/[email protected]/
Quoted 6 lines
For existing features, it might make some sense. But for new features, I don't think so. It defines the enabling order for new features that we must enable a feature for non-TDX VMs first and then TDs. And people might want to bypass this rule by abusing the
TDX_CFG_EXTRA_F()when only one line ofTDX_CFG_EXTRA_F()is enough to enable a feature for TDs but more effort required to enable it for non-TDX VMs.
Oh, you just remind me that I should add comment for TDX_CFG_EXTRA_F() that it must be used with a strong justification.
Quoted 2 lines
Maybe I miss somthing. I would like to see stronger reasons for such decision.
Hi Binbin,
Quoted 9 lines · Binbin Wu
Specifically, this series builds a KVM-side allowlist of supported TDX directly configurable CPUID bits to:
- FilterKVM_TDX_CAPABILITIES
Replace the hardcoded denylist to only report configurable bits that KVM explicitly supports.
- ValidateKVM_TDX_INIT_VM
Reject any configurable bit that the TDX module allows but KVM does not support, as well as CPUID entries with an unexpected subleaf.
Today's denylist only rejects TSX and WAITPKG. Everything else is allowed.
Obviously, the TDX module allows directly setting virtual CPUID values only for a subset of CPUID leaves, not all of them. So "everything else" above is that subset minus TSX and WAITPKG.
My question is: is there a feature in that "everything else" that causes host state clobbering today?
In other words, does this patch only build the infrastructure for addressing future clobbering issues, or does it also fix a specific bug?
Thanks!
**Disclaimer**: I am new to KVM and TDX, still learning, let me know if some of my comments are off.
It took me several days digging through docs and code to understand what is going on here, so let me summarize my understanding below and please correct me where I am wrong.
Quoted 9 lines · Binbin Wu
Hi,
The purpose of this patch series is to prevent userspace from enabling host state clobbering features that KVM does not support for TDX. A host state clobbering feature exposed on a new TDX module/platform can corrupt host state if KVM does not explicitly save and restore the related MSR(s) across host/guest transitions. If such a feature is blindly exposed to and used by a TD, the host will behave unexpectedly.
Quoted 5 lines
Except for a few fixed-1 bits required for basic TDX support, host state clobbering features are either directly configurable or gated by TD ATTRIBUTES/XFAM. So an allowlist covering only the directly configurable CPUID bits, plus the corresponding filtering and validation, is sufficient to serve the purpose while keeping the code footprint small.
So the general principle is:
- KVM is responsible for protecting the host state from being clobbered by the TD. Different approaches can be taken here.
- Saving and restoring the host state within KVM itself.
- For a subset of registers, relying on the TDX module to save and restore the host state.
- Disallowing certain TD features altogether if they pose a risk to host state.
- TDX module is responsible for saving and restoring TD state.
We have 2 boundaries: VMM <-> TDX module and TDX module <-> TD. Only the first one is relevant to host state clobbering.
Approach
========
Your patch-set adds an explicit allowlist to KVM: a TD may only use a feature if every register it needs is either saved/restored by KVM itself, or is preserved across TDH.VP.ENTER by the TDX module or HW.
I think this approach is safe and sound: each feature is checked before it is added to the allowlist. No surprises.
A CPU feature is associated with a group of registers, e.g. WAITPKG is the IA32_UMWAIT_CONTROL MSR. So instead of trying to control individual register access by TD, KVM controls which features are exposed to the TD.
CPUID is the actual mechanism for turning CPU features on/off for a TD. This patch works because the TDX module does not let a TD access the registers behind a CPU feature unless that feature is exposed via the virtual CPUID. An attempt to do so results in #UD or #GP.
Not every configurable CPUID field gates some registers, though. Some are pure enumeration, e.g. family/model/stepping and cache parameters.
So the approach is for KVM to look at what features are enabled in the virtualized CPUID, and reject the unsafe ones.
Details
=======
TDX module enables TD features only at TD build time, via TD_PARAMS structure of the TDH.MNG.INIT seamcall. This happens in the KVM_TDX_INIT_VM ioctl, after which the feature set cannot be widened. Migration cannot change them either: the TDH.IMPORT.STATE.IMMUTABLE seamcall imports the source TD's configuration state as-is.
The virtual CPUID values are built based on what the TDX module supports, what the host supports, and what userspace passed in via the KVM_TDX_INIT_VM ioctl.
The ioctl carries 'ATTRIBUTES', 'XFAM', and 'CPUID_CONFIG', all fields of 'TD_PARAMS'. ATTRIBUTES and XFAM turn features on/off and shape CPUID, but KVM only allows bits it already knows about, so there is no clobbering risk there.
CPUID_CONFIG is different: it lets userspace set CPUID leaf values directly (but not for any CPUID, only for pieces of it the TDX module allows, and KVM enumerates them via the KVM_TDX_CAPABILITIES ioctl).
That is the problematic part. KVM does not check its contents, beyond a denylist that only filters out TSX and WAITPKG features. Everything else is allowed.
So your patch-set basically kicks out the current denylist mechanism and replaces it with an allowlist of directly configurable CPUID leaves and bits.
There is a long list of CPUID pieces you allowed. I wanted to acknowledge that it must have been quite an effort to go through all of them and determine which ones are safe to allow.
Did I understand your work correctly?
Quoted 5 lines
Expected host state clobbering behavior for TDX
===============================================
We also want to call for discussions about the expected host state clobbering behavior for TDX here for future features.
Quoted 8 lines
For a normal VMX guest, VM entry/exit behavior for a given piece of CPU state is architecturally defined: state is either switched by hardware via VMCS host/guest fields, or left as the guest value on VM exit and managed by KVM in software.
For TDs, the host/guest transition goes through TDH.VP.ENTER, and what the TDX module does with a given piece of host state is defined by the TDX module ABI rather than by the x86 architecture.
I think I found this contract: the TDH.VP.ENTER definition in the ABI spec, section "CPU State Preservation Following a Successful TD Entry and a TD Exit". It refers to a table that lists the MSRs whose value may not be preserved across TD entry and exit, with the condition for each, e.g.
IA32_PL0_SSP Init(XFAM[11] | XFAM[12])
IA32_UMWAIT_CONTROL Init(virt. CPUID(7,0).ECX[5])
This is from an older version of the TDX ABI specification. I could not find 'msr_preservation.pdf' published. But I assume it is published.
Anyway, seems to be a clear contract to me.
Quoted 8 lines
What we would like to align on is the expected baseline behavior of TDH.VP.ENTER for future features. The proposal is to have TDX simply match VMX behavior, i.e. on return from TDH.VP.ENTER, state that VMX would restore from the VMCS host fields is restored, and state that VMX would leave as the guest value is clobbered. That keeps a single model for VMM, and means enabling a new feature for TDs requires the same work flow as enabling it for VMX.
Let me restate this to check I follow.
For a VMX guest, it is the VMCS host-state area that is used for restoring host state. KVM writes it in advance, hardware restores it upon VM exit.
Most of it is loaded unconditionally: control registers, RSP, RIP, SYSENTER MSRs and some more. But there are some MSRs that are restored only if KVM configures the corresponding control bits.
Example: IA32_PAT, IA32_EFER. If the control is clear, the MSR keeps the guest value upon VM exit.
In KVM some of those controls are set once, e.g. for IA32_PAT. Others are toggled at run time, e.g. "load IA32_EFER" and "load IA32_PERF_GLOBAL_CTRL". So there is dynamicity there.
So the proposal is that TDH.VP.ENTER should draw the same line: whatever VMX would reload from the host-state area is preserved, whatever VMX leaves as the guest value is clobbered and KVM handles it. Is that right?
So with TDX module there is a twist. First of all, my understanding is that the SEAM VMCS (which controls VMM <-> TDX module state save/restore) is configured when the TDX module is loaded and stays fixed after that. So no dynamicity there.
Second, the SDM says SEAMCALL behaves like an SMM VM exit and SEAMRET like a VM entry returning from SMM (SDM 35.1.1, 35.1.2), and SMM VM exits save state into the guest-state area (SDM 34.15.2.2).
For reference: An SMM VM exit is a VM exit that begins outside SMM and that ends in SMM.
So if I understand correctly, for VMM <-> TDX module switches, the hardware uses the VMCS guest-state area, not the host-state area. Not like in VMX guest case. On TDH.VP.ENTER it saves the VMM state to the guest-state area, and on return it restores the VMM state.
IOW, looks like there may be significant differences between the VMM <-> TDX module state handling and VMX VM entries and exits.
So maybe the way to go is to just follow 'msr_preservation.pdf' and adjust the allowlist? I find this approach safe and acceptable.
Artem.
Edgecombe, Rick P
maintainerRe: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Xiaoyao LiQuoted 5 lines · Xiaoyao Li
Quoted 4 lines
In this version,
TDX_CFG_F()already check against kvm_cpu_caps[], if these features
are not in kvm_cpu_caps[], it will not be exposed to userspace anyway.
Quoted 7 lines
I still think the reasoning that we omit them because they are not contained in kvm_cpu_caps[] sounds not right. Based on it, so when we are going to add a new feature for TDX, we need to first manually check the KVM code to see if that feature is contained in kvm_cpu_caps[] already. If not, we just don't add it to TDX's list. Then why need to cap the result kvm_cpu_caps[] for
TDX_CFG_F()again? Just for safety in case human make mistake and misread the code of kvm_cpu_caps[]?
We could warn on the condition and not just silently strip it.
Quoted 15 lines
I think they are two independent steps:
1. list the CPUID features that KVM can support for TDs.
2. apply additional restrictions, e.g., if a feature is not allowed for non-TDX VMs, it cannot be allowed for TDs.Quoted 8 lines
Quoted 5 lines
In the end, they might be disallowed to be configured to TDs because KVM doesn't allow them for VMX VMs. This is also the point I want to discuss. Do we really want to make such restriction that KVM cannot enable/allow a feature for TDs unless KVM first enables/allows it for VMX VMs? What's reason behind it?
Sean mentioned it that "generally speaking, KVM shouldn't allow features that KVM doesn't support for non-TDX VMs" in
https://lore.kernel.org/kvm/[email protected]/
Quoted 8 lines
For existing features, it might make some sense. But for new features, I don't think so. It defines the enabling order for new features that we must enable a feature for non-TDX VMs first and then TDs. And people might want to bypass this rule by abusing the
TDX_CFG_EXTRA_F()when only one line ofTDX_CFG_EXTRA_F()is enough to enable a feature for TDs but more effort required to enable it for non-TDX VMs.Maybe I miss somthing. I would like to see stronger reasons for such decision.
I think "generally speaking" means, it's not a hard rule.
As for why to prefer it, I think we would normally want regulars VMs and TDs to work similarly. Especially those that have some of the virtualization handled by KVM. But TDX module's behavior of a feature can conform to KVM's only if KVM's already exists. Take for example split lock detection. The normal VM KVM support initially went through several iterations of design. Separately, TDX ended up with a different solution. Imagine if we had enabled the TDX arch one, before solving the general KVM problems. Then we would end up with two different behaviors, or a worse KVM behavior as it tries to conform to TDX module's behavior.
So we need to at least solve a feature at that level before deciding KVM's handling of it. This could be done while enabling the feature for TDX only, but often would involve solving the problems for normal VMs too. In the end, it's the generic KVM behavior that needs to be solved before enabling the feature.
Ideally we could consider normal VM and TD at the same time. I expect we will start doing that after this series is in place. Not a hard rule, but a norm.
Edgecombe, Rick P
maintainerRe: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Xiaoyao LiQuoted 2 lines · Xiaoyao Li
yeah. This changes KVM's behavior and we definitely need to call it out, and provide justification.
Agreed. We need to call out clearly the uABI changes. But we don't need to be paranoid if there is a path to fix it if we get it wrong.
Edgecombe, Rick P
maintainerQuoted 4 lines · Artem Bityutskiy
**Disclaimer**: I am new to KVM and TDX, still learning, let me know if some of my comments are off.
...
Quoted 3 lines
So maybe the way to go is to just follow 'msr_preservation.pdf' and adjust the allowlist? I find this approach safe and acceptable.
We are kind of discussing what recommendations we should give about how msr_preservation.pdf should be defined for new CPUID bit based features. So saying to follow msr_preservation.pdf is self referential.
Again, please do not treat the TDX specs as something to be handed down and "followed". I think this is something to get used to for TDX. I mean, upstream never wants to adapt to platform arch that fits awkwardly, but it's even tougher to swallow when the arch is mostly SW defined. And further, the people working on the TDX arch want to hear such requirements from upstream.
Here, the thing to discuss is how TDX should define new features that will clobber host state (e.g. bits that would appear in msr_preservation.pdf as not being preserved).
There have been several PUCK discussions on the problem this series is tackling, and actually several attempts to solve the problem during the base series. More recently a host clobbering control was proposed that attempted to make it safe, but it was not accepted. That proposal brought up the topic of whether having select states clobbered was actually an unproven optimization.
Now that we are moving back to an allow list type solution, what guidance should we give on this other surfaced topic. Since TDX shares some save/restore logic with normal VMs, we should have it work well with that code. So forgetting about the performance optimization question, how to have it work in a sensible way with the shared code paths.
Quoted 5 lines · Artem Bityutskiy
**Disclaimer**: I am new to KVM and TDX, still learning, let me know if some of my comments are off.
It took me several days digging through docs and code to understand what is going on here,
Thanks!
[...]
Quoted 13 lines
CPUID_CONFIGis different: it lets userspace set CPUID leaf values directly (but not for any CPUID, only for pieces of it the TDX module allows, and KVM enumerates them via theKVM_TDX_CAPABILITIESioctl).That is the problematic part. KVM does not check its contents, beyond a denylist that only filters out TSX and WAITPKG features. Everything else is allowed.
So your patch-set basically kicks out the current denylist mechanism and replaces it with an allowlist of directly configurable CPUID leaves and bits.
[...]
Quoted 2 lines
Did I understand your work correctly?
Yes, you understand it correctly.
Quoted 21 lines
Quoted 6 lines
Expected host state clobbering behavior for TDX
===============================================
We also want to call for discussions about the expected host state clobbering behavior for TDX here for future features.Quoted 8 lines
For a normal VMX guest, VM entry/exit behavior for a given piece of CPU state is architecturally defined: state is either switched by hardware via VMCS host/guest fields, or left as the guest value on VM exit and managed by KVM in software.
For TDs, the host/guest transition goes through TDH.VP.ENTER, and what the TDX module does with a given piece of host state is defined by the TDX module ABI rather than by the x86 architecture.
I think I found this contract: the TDH.VP.ENTER definition in the ABI spec, section "CPU State Preservation Following a Successful TD Entry and a TD Exit". It refers to a table that lists the MSRs whose value may not be preserved across TD entry and exit, with the condition for each, e.g.
Quoted 5 lines
IA32_PL0_SSPInit(XFAM[11] | XFAM[12])
IA32_UMWAIT_CONTROLInit(virt. CPUID(7,0).ECX[5])This is from an older version of the TDX ABI specification. I could not find 'msr_preservation.pdf' published. But I assume it is published.
You can find these tables from the "Intel TDX Module ABI Definitions" part on
https://www.intel.com/content/www/us/en/developer/tools/trust-domain-extensions/documentation.html
[...]
Quoted 3 lines
So maybe the way to go is to just follow 'msr_preservation.pdf' and adjust the allowlist? I find this approach safe and acceptable.
Rick explained the long history:
https://lore.kernel.org/all/[email protected]/
The Denylist based solution is not a clean solution:
- It couples the feature enabling for normal VMs with TDX tightly.
- Each time a new feature is added in the denylist, it needs to be backported to old KVM versions
Quoted 1 line
Artem.
Quoted 19 lines · Artem Bityutskiy
Hi Binbin,
Quoted 9 lines · Binbin Wu
Specifically, this series builds a KVM-side allowlist of supported TDX directly configurable CPUID bits to:
- FilterKVM_TDX_CAPABILITIES
Replace the hardcoded denylist to only report configurable bits that KVM explicitly supports.
- ValidateKVM_TDX_INIT_VM
Reject any configurable bit that the TDX module allows but KVM does not support, as well as CPUID entries with an unexpected subleaf.Today's denylist only rejects TSX and WAITPKG. Everything else is allowed.
Obviously, the TDX module allows directly setting virtual CPUID values only for a subset of CPUID leaves, not all of them. So "everything else" above is that subset minus TSX and WAITPKG.
Quoted 6 lines
My question is: is there a feature in that "everything else" that causes host state clobbering today?
In other words, does this patch only build the infrastructure for addressing future clobbering issues, or does it also fix a specific bug?
The Denylist works fine today. But the TDX module evolves. There are new host state clobbering features coming, e.g. FRED.
The Denylist based solution is not a clean solution:
- It couples the feature enabling for normal VMs with TDX tightly.
- Each time a new feature is added in the denylist, it needs to be backported to old KVM versions
Quoted 2 lines
Thanks!
Quoted 14 lines · Rick P
Quoted 5 lines · Xiaoyao Li
Quoted 4 lines · Binbin Wu
Quoted 4 lines
Quoted 4 lines
So if the approach in this series is taken, I think we still need such an opt- in interface to tell the TDX module that the VMM is now filtering the CPUID bits so that the TDX module knows that it's safe to report new host state clobbering features.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Yep. Can we think about what it would look like? Easiest would be a bit passed in
TDH_SYS_CONFIG. But thenarch/x86is saying how KVM will behave. Ok to me, for the simplicity. Could come with a nice comment.Can you just treat current KVM behavior of allowing userspace to enable any configurable bits as the bug of KVM and backport this series as Binbin suggested below? Instead of introducing more opt-in knobs.
Quoted 7 lines
Ok, so if we are agreed on the other branch of the thread, the only big question
is: Do we want an opt-in for future clobbering CPUID bits.I think either is ok. I don't love the precedent that we asserted that no new clobber bits could be added without opt-in, and then we would backport changes to allow this anyway. But on pure code, the backport would be simpler in the long term. If you guys are strongly in favor, I can agree.
Had a discussion with Rick off list.
We can defer the opt-in design or backport decision because the patches will still be correct and an improvement in any case.
Quoted 5 lines · Binbin Wu
Quoted 4 lines
So maybe the way to go is to just follow 'msr_preservation.pdf' and adjust the allowlist? I find this approach safe and acceptable.
Quoted 7 lines
Rick explained the long history:
https://lore.kernel.org/all/[email protected]/The Denylist based solution is not a clean solution:
- It couples the feature enabling for normal VMs with TDX tightly.
- Each time a new feature is added in the denylist, it needs to be backported to old KVM versions
Please ignore this part since I replied to the wrong thread.
Quoted 3 lines
Quoted 2 lines
Artem.
Quoted 3 lines · Rick P
We are kind of discussing what recommendations we should give about how msr_preservation.pdf should be defined for new CPUID bit based features. So saying to follow msr_preservation.pdf is self referential.
OK, thanks for elaborating. The e-mail was vague about that.
Quoted 5 lines
Again, please do not treat the TDX specs as something to be handed down and "followed". I think this is something to get used to for TDX. I mean, upstream never wants to adapt to platform arch that fits awkwardly, but it's even tougher to swallow when the arch is mostly SW defined. And further, the people working on the TDX arch want to hear such requirements from upstream.
Agreed, that matches my understanding. Reminders are useful in general, but this was not that case.
Quoted 10 lines
Here, the thing to discuss is how TDX should define new features that will clobber host state (e.g. bits that would appear in msr_preservation.pdf as not being preserved).
There have been several PUCK discussions on the problem this series is tackling, and actually several attempts to solve the problem during the base series. More recently a host clobbering control was proposed that attempted to make it safe, but it was not accepted. That proposal brought up the topic of whether having select states clobbered was actually an unproven optimization.
Quoted 5 lines
Now that we are moving back to an allow list type solution, what guidance should we give on this other surfaced topic. Since TDX shares some save/restore logic with normal VMs, we should have it work well with that code. So forgetting about the performance optimization question, how to have it work in a sensible way with the shared code paths.
Rick,
In general, the TDX case and the VMX case behaving the same way is best. I thought that consistency is always obviously a good thing, but let me acknowledge it explicitly.
I was trying to dig deeper into the proposal and analyze it.
<cite>
The proposal is to have TDX simply match VMX behavior, i.e. on return from TDH.VP.ENTER, state that VMX would restore from the VMCS host fields is restored, and state that VMX would leave as the guest value is clobbered. That keeps a single model for VMM, and means enabling a new feature for TDs requires the same work flow as enabling it for VMX. </cite>
My point was that hardware behaves differently for VMX guests and for the VMM<->TDX. This is not about following "boss specs", it is what the SDM describes.
For VMX, the VMCS host-state area is loaded by hardware on VM exit (SDM 27.5). For SEAMCALL and SEAMRET, the SDM seems to say they operate like an SMM VM exit and a VM entry returning from SMM (SDM 35.1), and those save state into the guest-state area of the transfer VMCS (SDM 34.15.2.2).
VMX:
VM entry = load guest state from guest-state area VM exit = save guest state into guest-state area,
load host state from host-state area
TDX:
SEAMCALL = save host state into SEAM VMCS guest-state area,
load module state from SEAM VMCS host-state area
SEAMRET = restore host state from SEAM VMCS guest-state area
I may be reading the SDM wrong, let me know.
So there are differences, and I was hoping to:
- Be corrected if I misinterpret the SDM and how things work.
- Get comments on whether the proposal took this into account.
- Get comments on how this affects, or does not affect, the proposal.
Thanks, Artem.
Quoted 63 lines · Artem Bityutskiy
Quoted 3 lines · Rick P
We are kind of discussing what recommendations we should give about how msr_preservation.pdf should be defined for new CPUID bit based features. So saying to follow msr_preservation.pdf is self referential.
OK, thanks for elaborating. The e-mail was vague about that.
Quoted 5 lines
Again, please do not treat the TDX specs as something to be handed down and "followed". I think this is something to get used to for TDX. I mean, upstream never wants to adapt to platform arch that fits awkwardly, but it's even tougher to swallow when the arch is mostly SW defined. And further, the people working on the TDX arch want to hear such requirements from upstream.
Agreed, that matches my understanding. Reminders are useful in general, but this was not that case.
Quoted 10 lines
Here, the thing to discuss is how TDX should define new features that will clobber host state (e.g. bits that would appear in msr_preservation.pdf as not being preserved).
There have been several PUCK discussions on the problem this series is tackling, and actually several attempts to solve the problem during the base series. More recently a host clobbering control was proposed that attempted to make it safe, but it was not accepted. That proposal brought up the topic of whether having select states clobbered was actually an unproven optimization.
Quoted 5 lines
Now that we are moving back to an allow list type solution, what guidance should we give on this other surfaced topic. Since TDX shares some save/restore logic with normal VMs, we should have it work well with that code. So forgetting about the performance optimization question, how to have it work in a sensible way with the shared code paths.
Rick,
In general, the TDX case and the VMX case behaving the same way is best. I thought that consistency is always obviously a good thing, but let me acknowledge it explicitly.
I was trying to dig deeper into the proposal and analyze it.
<cite>
The proposal is to have TDX simply match VMX behavior, i.e. on return from TDH.VP.ENTER, state that VMX would restore from the VMCS host fields is restored, and state that VMX would leave as the guest value is clobbered. That keeps a single model for VMM, and means enabling a new feature for TDs requires the same work flow as enabling it for VMX. </cite>My point was that hardware behaves differently for VMX guests and for the VMM<->TDX. This is not about following "boss specs", it is what the SDM describes.
For VMX, the VMCS host-state area is loaded by hardware on VM exit (SDM 27.5). For SEAMCALL and SEAMRET, the SDM seems to say they operate like an SMM VM exit and a VM entry returning from SMM (SDM 35.1), and those save state into the guest-state area of the transfer VMCS (SDM 34.15.2.2).
VMX:
VM entry = load guest state from guest-state area VM exit = save guest state into guest-state area,
load host state from host-state area
Quoted 6 lines
TDX:
SEAMCALL = save host state into SEAM VMCS guest-state area,
load module state from SEAM VMCS host-state area
SEAMRET = restore host state from SEAM VMCS guest-state areaI may be reading the SDM wrong, let me know.
That's my understanding too.
Quoted 5 lines
So there are differences, and I was hoping to:
- Be corrected if I misinterpret the SDM and how things work.
- Get comments on whether the proposal took this into account.
- Get comments on how this affects, or does not affect, the proposal.
For host state clobbering behavior, we cares about the values of the host (VMX root mode) after SEAMRET.
When there is a control/field for "load host state from host-state area", I think there are two cases:
- If there is the corresponding control/field for "load guest state from guest-state area", the TDX module could leverage it.
- If there is no such corresponding control/field for "load guest state from guest-state area", the TDX module could do it in software way to mimic it.
So from the view of the VMM, it can have the aligned behavior on host state clobbering behavior.
Quoted 5 lines · Binbin Wu
Quoted 5 lines
VMX:
VM entry = load guest state from guest-state area VM exit = save guest state into guest-state area,
load host state from host-state area
Quoted 8 lines
Quoted 6 lines
TDX:
SEAMCALL = save host state into SEAM VMCS guest-state area,
load module state from SEAM VMCS host-state area
SEAMRET = restore host state from SEAM VMCS guest-state areaI may be reading the SDM wrong, let me know.
That's my understanding too.
Good, thanks for confirming.
Quoted 16 lines
Quoted 5 lines
So there are differences, and I was hoping to:
- Be corrected if I misinterpret the SDM and how things work.
- Get comments on whether the proposal took this into account.
- Get comments on how this affects, or does not affect, the proposal.For host state clobbering behavior, we cares about the values of the host (VMX root mode) after SEAMRET.
When there is a control/field for "load host state from host-state area", I think there are two cases:
- If there is the corresponding control/field for "load guest state from guest-state area", the TDX module could leverage it.
- If there is no such corresponding control/field for "load guest state from guest-state area", the TDX module could do it in software way to mimic it.
Quoted 2 lines
So from the view of the VMM, it can have the aligned behavior on host state clobbering behavior.
Now I see what you mean: make msr_preservation.pdf follow the same rule as the VMX host-state restore, and let the TDX module help where HW behaves differently (call this SW restore vs HW restore via VMCS).
That sounds good to me.
My only doubt is whether it can be guaranteed in every case. A HW restore happens after a SW restore.
E.g., IA32_DEBUGCTL - HW clears it on VM exit (SDM 30.5.1), so whatever TDX module puts there on the exit path, will be overwritten. Not that this is an issue today, just using this as an example.
But I'd guess there would be only few problematic cases (if any).
Then you wrote this:
<cite>
FRED is a useful concrete example. Under VMX, the FRED host state in IA32_FRED_CONFIG, IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX module is expected to restore these MSRs on TDH.VP.ENTER return. IA32_FRED_RSP0 and IA32_PL0_SSP (a.k.a. IA32_FRED_SSP0) are handled by software, so the TDX module is expected to clobber them on TDH.VP.ENTER return. </cite>
That one reads as obviously right to me. If VMX and TDX differed in how IA32_FRED_RSP0 and IA32_PL0_SSP are handled, that would be a red flag.
Did you go through all the MSRs and check that the VMX and TDX behavior matches today?
Thanks!
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Edgecombe, Rick PQuoted 28 lines · Rick P
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
In the end, they might be disallowed to be configured to TDs because KVM doesn't allow them for VMX VMs. This is also the point I want to discuss. Do we really want to make such restriction that KVM cannot enable/allow a feature for TDs unless KVM first enables/allows it for VMX VMs? What's reason behind it?
Sean mentioned it that "generally speaking, KVM shouldn't allow features that KVM doesn't support for non-TDX VMs" in
https://lore.kernel.org/kvm/[email protected]/Quoted 8 lines
For existing features, it might make some sense. But for new features, I don't think so. It defines the enabling order for new features that we must enable a feature for non-TDX VMs first and then TDs. And people might want to bypass this rule by abusing the
TDX_CFG_EXTRA_F()when only one line ofTDX_CFG_EXTRA_F()is enough to enable a feature for TDs but more effort required to enable it for non-TDX VMs.Maybe I miss somthing. I would like to see stronger reasons for such decision.
I think "generally speaking" means, it's not a hard rule.
As for why to prefer it, I think we would normally want regulars VMs and TDs to work similarly. Especially those that have some of the virtualization handled by KVM. But TDX module's behavior of a feature can conform to KVM's only if KVM's already exists. Take for example split lock detection. The normal VM KVM support initially went through several iterations of design. Separately, TDX ended up with a different solution. Imagine if we had enabled the TDX arch one, before solving the general KVM problems. Then we would end up with two different behaviors, or a worse KVM behavior as it tries to conform to TDX module's behavior.
Quoted 7 lines
So we need to at least solve a feature at that level before deciding KVM's handling of it. This could be done while enabling the feature for TDX only, but often would involve solving the problems for normal VMs too. In the end, it's the generic KVM behavior that needs to be solved before enabling the feature.
Ideally we could consider normal VM and TD at the same time. I expect we will start doing that after this series is in place. Not a hard rule, but a norm.
I don't think "split lock detection" is a good example. We are discussing virtualizing a feature, or allowing a feature to be exposed to non-TDX and TDX guests. While "split lock detection" is not a virtualizable feature and how KVM handles it is all about how KVM fixes the architectural flaw of it.
Generally, I agree with your point that we need to think at higher level before KVM deciding to support virtualizing a feature. But after the generic level consideration is done, there can be different orders:
1. allow a feature for non-TDX VMs and TDs at same time. i.e., in one patch or in a single series.
2. allow a feature for non-TDX VMs first and then TDs. This can be due to by that time TDX's spec for the feature has not been defined/finalized.
3. allow a feature for TDs first and then non-TDX VMs. This can be due to it requires more effort/patches to support virtualizing a feature for non-TDX VMs.
Usually allowing a feature for TDs requires far less efforts than for non-TDX VMs, because most of the virtualization work is done by TDX module. E.g., for TDs, KVM usually only needs to add code to restore the host states that are not restored by TDX module. However for non-TDX VMs, KVM needs to do more: emulate the architectural behavior of the MSRs, context switch states, and the nested handling, etc. It's likely that a series to enable a feature for non-TDX VMs spends several Linux releases to finally get merged. Making TDX support depends on it seems not that necessary.
Sean Christopherson
maintainerRe: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
Quoted 16 lines · Xiaoyao Li
Quoted 15 lines · Rick P
Quoted 15 lines
Quoted 8 lines
Quoted 5 lines
In the end, they might be disallowed to be configured to TDs because KVM doesn't allow them for VMX VMs. This is also the point I want to discuss. Do we really want to make such restriction that KVM cannot enable/allow a feature for TDs unless KVM first enables/allows it for VMX VMs? What's reason behind it?
Sean mentioned it that "generally speaking, KVM shouldn't allow features that KVM doesn't support for non-TDX VMs" in
https://lore.kernel.org/kvm/[email protected]/For existing features, it might make some sense. But for new features, I don't think so. It defines the enabling order for new features that we must enable a feature for non-TDX VMs first and then TDs. And people might want to bypass this rule by abusing the
TDX_CFG_EXTRA_F()when only one line ofTDX_CFG_EXTRA_F()is enough to enable a feature for TDs but more effort required to enable it for non-TDX VMs.
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Maybe I miss somthing. I would like to see stronger reasons for such decision.
I think "generally speaking" means, it's not a hard rule.
Ya.
Quoted 10 lines
Quoted 10 lines
As for why to prefer it, I think we would normally want regulars VMs and TDs to work similarly. Especially those that have some of the virtualization handled by KVM. But TDX module's behavior of a feature can conform to KVM's only if KVM's already exists. Take for example split lock detection. The normal VM KVM support initially went through several iterations of design. Separately, TDX ended up with a different solution. Imagine if we had enabled the TDX arch one, before solving the general KVM problems. Then we would end up with two different behaviors, or a worse KVM behavior as it tries to conform to TDX module's behavior.
Quoted 7 lines
Quoted 7 lines
So we need to at least solve a feature at that level before deciding KVM's handling of it. This could be done while enabling the feature for TDX only, but often would involve solving the problems for normal VMs too. In the end, it's the generic KVM behavior that needs to be solved before enabling the feature.
Ideally we could consider normal VM and TD at the same time. I expect we will start doing that after this series is in place. Not a hard rule, but a norm.
Eh, I'm with Xiaoyao. Yeah, *ideally* we'd magically enable everything everywhere all at once. In reality, different VM types are going to support features at different times. More importantly, as Xiaoyao points out below in #1, unless we enable everyting in a single patch, which is probably a terrible idea in most cases, we'll still end up with staged/progressive enabling, i.e. we still need to have patches that selectively enable and advertise a feature only for the VM types that actually support the feature.
This is all quite similar to Intel and AMD feature enabling being done at different times. The biggest difference is that Intel and AMD are mutually exclusive and so KVM_GET_SUPPORTED_CPUID always reports the correct information, but TDX already provides KVM_TDX_CAPABILITIES, so AFAICT we still get accurate reporting for TDX, just in a slightly different way.
Quoted 4 lines
I don't think "split lock detection" is a good example. We are discussing virtualizing a feature, or allowing a feature to be exposed to non-TDX and TDX guests. While "split lock detection" is not a virtualizable feature and how KVM handles it is all about how KVM fixes the architectural flaw of it.
Yep. If there are actual decisions to be made, versus simply adhering to the architecture, then we'll need to incorporate the needs/abilities of flavors of VMs KVM supports. But for feature virtualization where right vs. wrong is dictated by hardware specs, there really isn't anything we can do in KVM to affect the guest-visible behavior (beyond things like performance characteristics, but those aren't ABI in any case).
Quoted 21 lines
Generally, I agree with your point that we need to think at higher level before KVM deciding to support virtualizing a feature. But after the generic level consideration is done, there can be different orders:
1. allow a feature for non-TDX VMs and TDs at same time. i.e., in one patch or in a single series.
2. allow a feature for non-TDX VMs first and then TDs. This can be due to by that time TDX's spec for the feature has not been defined/finalized.
3. allow a feature for TDs first and then non-TDX VMs. This can be due to it requires more effort/patches to support virtualizing a feature for non-TDX VMs.
Usually allowing a feature for TDs requires far less efforts than for non-TDX VMs, because most of the virtualization work is done by TDX module. E.g., for TDs, KVM usually only needs to add code to restore the host states that are not restored by TDX module. However for non-TDX VMs, KVM needs to do more: emulate the architectural behavior of the MSRs, context switch states, and the nested handling, etc. It's likely that a series to enable a feature for non-TDX VMs spends several Linux releases to finally get merged. Making TDX support depends on it seems not that necessary.
Edgecombe, Rick P
maintainerRe: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
Quoted 1 line · Sean Christopherson
Eh, I'm with Xiaoyao.
I'm not sure what the disagreement is actually. No one is saying *never* TD first I think? Everyone agrees it is ideal to solve normal VMs before settling the TDX behavior.
Quoted 8 lines
Yeah, *ideally* we'd magically enable everything everywhere all at once. In reality, different VM types are going to support features at different times. More importantly, as Xiaoyao points out below in #1, unless we enable everyting in a single patch, which is probably a terrible idea in most cases, we'll still end up with staged/progressive enabling, i.e. we still need to have patches that selectively enable and advertise a feature only for the VM types that actually support the feature.
Quoted 5 lines
This is all quite similar to Intel and AMD feature enabling being done at different times. The biggest difference is that Intel and AMD are mutually exclusive and so
KVM_GET_SUPPORTED_CPUIDalways reports the correct information, but TDX already providesKVM_TDX_CAPABILITIES, so AFAICT we still get accurate reporting for TDX, just in a slightly different way.
I think this actually surfaces another problem with TD-first enabling. KVM_TDX_CAPABILITIES only returns the directly configurable bits. Then recall, KVM_TDX_GET_CPUID returns the actual TDX module's view of CPUID bits to userspace. Then userspace calls KVM_SET_CPUID to actually put them on KVM's vcpu so they can match between Qemu, KVM and TDX
So if a bit is enabled for KVM_TDX_CAPABILITIES, but not yet in KVM_GET_SUPPORTED_CPUID. How should userspace interpret KVM_GET_SUPPORTED_CPUID? It can ignore it for TDX, but that is how it can find the PV bits today.
If we have a TD first feature, it could be a documentation update on how to interpret it. Or we could stuff the PV bits somewhere else for TDX and say to ignore KVM_GET_SUPPORTED_CPUID for TDX. I think we don't need to solve it before we begin filtering like this series has.
Quoted 6 lines
Quoted 5 lines
I don't think "split lock detection" is a good example. We are discussing virtualizing a feature, or allowing a feature to be exposed to non-TDX and TDX guests. While "split lock detection" is not a virtualizable feature and how KVM handles it is all about how KVM fixes the architectural flaw of it.
Quoted 6 lines
Yep. If there are actual decisions to be made, versus simply adhering to the architecture, then we'll need to incorporate the needs/abilities of flavors of VMs KVM supports. But for feature virtualization where right vs. wrong is dictated by hardware specs, there really isn't anything we can do in KVM to affect the guest-visible behavior (beyond things like performance characteristics, but those aren't ABI in any case).
(Copying from my now aborted response).
I think it is not just the the behavior to the guest that matters. I agree guest behavior should normally be settled by the bare metal arch. But it is also how the TDX module decides to make the host side behave. And how it ends up fitting in to KVM for normal VMs. For "normal bits" it doesn't matter too much. But for anything with extra TDX arch decisions on top, ideally the TDX arch would not be finalized before the normal VM KVM design got to some level of maturity.
In any case, I don't see any big disagreement between anyone actually. If anyone has any big worry please make it clear.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
Quoted 21 lines · Rick P
Quoted 1 line · Sean Christopherson
Eh, I'm with Xiaoyao.
I'm not sure what the disagreement is actually. No one is saying *never* TD first I think? Everyone agrees it is ideal to solve normal VMs before settling the TDX behavior.
Quoted 8 lines
Yeah, *ideally* we'd magically enable everything everywhere all at once. In reality, different VM types are going to support features at different times. More importantly, as Xiaoyao points out below in #1, unless we enable everyting in a single patch, which is probably a terrible idea in most cases, we'll still end up with staged/progressive enabling, i.e. we still need to have patches that selectively enable and advertise a feature only for the VM types that actually support the feature.
Quoted 5 lines
This is all quite similar to Intel and AMD feature enabling being done at different times. The biggest difference is that Intel and AMD are mutually exclusive and so
KVM_GET_SUPPORTED_CPUIDalways reports the correct information, but TDX already providesKVM_TDX_CAPABILITIES, so AFAICT we still get accurate reporting for TDX, just in a slightly different way.
Quoted 5 lines
I think this actually surfaces another problem with TD-first enabling.
KVM_TDX_CAPABILITIESonly returns the directly configurable bits. Then recall,KVM_TDX_GET_CPUIDreturns the actual TDX module's view of CPUID bits to userspace. Then userspace callsKVM_SET_CPUIDto actually put them on KVM's vcpu so they can match between Qemu, KVM and TDX
That brings up a point..
Today, vcpu->arch.cpu_caps[] is capped by kvm_cpu_caps[] (plus a few special cases). As mentioned in the cover letter, this patch series doesn't enforce consistency between KVM's view and the guest's view of vCPU capabilities because KVM doesn't currently use its own view to make decisions for TDs (e.g. saving/restoring feature-related MSRs).
However, if KVM starts making decisions for TDX based on vcpu->arch.cpu_caps[], intersecting userspace input with kvm_cpu_caps[] will not work for TDX. I think this is probably needed in the future? If so, allowing features outside of kvm_cpu_caps[] for TDX means we will need TDX-specific handling to construct KVM's view of vCPU capabilities. That likely implies tracking all known/supported TDX features, which is doable, but it will make the allow list bigger.
Quoted 34 lines
So if a bit is enabled for
KVM_TDX_CAPABILITIES, but not yet inKVM_GET_SUPPORTED_CPUID. How should userspace interpretKVM_GET_SUPPORTED_CPUID? It can ignore it for TDX, but that is how it can find the PV bits today.If we have a TD first feature, it could be a documentation update on how to interpret it. Or we could stuff the PV bits somewhere else for TDX and say to ignore
KVM_GET_SUPPORTED_CPUIDfor TDX. I think we don't need to solve it before we begin filtering like this series has.Quoted 6 lines
Quoted 5 lines
I don't think "split lock detection" is a good example. We are discussing virtualizing a feature, or allowing a feature to be exposed to non-TDX and TDX guests. While "split lock detection" is not a virtualizable feature and how KVM handles it is all about how KVM fixes the architectural flaw of it.
Quoted 6 lines
Yep. If there are actual decisions to be made, versus simply adhering to the architecture, then we'll need to incorporate the needs/abilities of flavors of VMs KVM supports. But for feature virtualization where right vs. wrong is dictated by hardware specs, there really isn't anything we can do in KVM to affect the guest-visible behavior (beyond things like performance characteristics, but those aren't ABI in any case).
(Copying from my now aborted response).
I think it is not just the the behavior to the guest that matters. I agree guest behavior should normally be settled by the bare metal arch. But it is also how the TDX module decides to make the host side behave. And how it ends up fitting in to KVM for normal VMs. For "normal bits" it doesn't matter too much. But for anything with extra TDX arch decisions on top, ideally the TDX arch would not be finalized before the normal VM KVM design got to some level of maturity.
In any case, I don't see any big disagreement between anyone actually. If anyone has any big worry please make it clear.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Edgecombe, Rick PQuoted 20 lines · Rick P
Quoted 8 lines · Sean Christopherson
Yeah, *ideally* we'd magically enable everything everywhere all at once. In reality, different VM types are going to support features at different times. More importantly, as Xiaoyao points out below in #1, unless we enable everyting in a single patch, which is probably a terrible idea in most cases, we'll still end up with staged/progressive enabling, i.e. we still need to have patches that selectively enable and advertise a feature only for the VM types that actually support the feature.
Quoted 5 lines
This is all quite similar to Intel and AMD feature enabling being done at different times. The biggest difference is that Intel and AMD are mutually exclusive and so
KVM_GET_SUPPORTED_CPUIDalways reports the correct information, but TDX already providesKVM_TDX_CAPABILITIES, so AFAICT we still get accurate reporting for TDX, just in a slightly different way.I think this actually surfaces another problem with TD-first enabling.
KVM_TDX_CAPABILITIESonly returns the directly configurable bits. Then recall,KVM_TDX_GET_CPUIDreturns the actual TDX module's view of CPUID bits to userspace. Then userspace callsKVM_SET_CPUIDto actually put them on KVM's vcpu so they can match between Qemu, KVM and TDX
Quoted 7 lines
So if a bit is enabled for
KVM_TDX_CAPABILITIES, but not yet inKVM_GET_SUPPORTED_CPUID. How should userspace interpretKVM_GET_SUPPORTED_CPUID? It can ignore it for TDX, but that is how it can find the PV bits today.> If we have a TD first feature, it could be a documentation update on how to interpret it. Or we could stuff the PV bits somewhere else for TDX and say to ignoreKVM_GET_SUPPORTED_CPUIDfor TDX. I think we don't need to solve it before we begin filtering like this series has.
The PV bits reported in KVM_GET_SUPPORTED_CPUID are the bits supported for non-TDX VMs. Only some of them are actually supported for TDX. I would suggest reporting TDX supported PV CPUIDs in KVM_TDX_CAPABILITIES as well.
Quoted 40 lines · Artem Bityutskiy
Quoted 5 lines · Binbin Wu
Quoted 5 lines
VMX:
VM entry = load guest state from guest-state area VM exit = save guest state into guest-state area,
load host state from host-state areaQuoted 8 lines
Quoted 6 lines
TDX:
SEAMCALL = save host state into SEAM VMCS guest-state area,
load module state from SEAM VMCS host-state area
SEAMRET = restore host state from SEAM VMCS guest-state areaI may be reading the SDM wrong, let me know.
That's my understanding too.
Good, thanks for confirming.
Quoted 16 lines
Quoted 5 lines
So there are differences, and I was hoping to:
- Be corrected if I misinterpret the SDM and how things work.
- Get comments on whether the proposal took this into account.
- Get comments on how this affects, or does not affect, the proposal.For host state clobbering behavior, we cares about the values of the host (VMX root mode) after SEAMRET.
When there is a control/field for "load host state from host-state area", I think there are two cases:
- If there is the corresponding control/field for "load guest state from guest-state area", the TDX module could leverage it.
- If there is no such corresponding control/field for "load guest state from guest-state area", the TDX module could do it in software way to mimic it.Quoted 2 lines
So from the view of the VMM, it can have the aligned behavior on host state clobbering behavior.
Now I see what you mean: make msr_preservation.pdf follow the same rule as the VMX host-state restore, and let the TDX module help where HW behaves differently (call this SW restore vs HW restore via VMCS).
Quoted 8 lines
That sounds good to me.
My only doubt is whether it can be guaranteed in every case. A HW restore happens after a SW restore.
E.g.,
IA32_DEBUGCTL- HW clears it on VM exit (SDM 30.5.1), so whatever TDX module puts there on the exit path, will be overwritten. Not that this is an issue today, just using this as an example.
But SEAMRET is actually a VM Entry, I think these special cases are during VM Exit. For a VM entry, the TDX module should be able to set whatever valid values for host.
Quoted 14 lines
But I'd guess there would be only few problematic cases (if any).
Then you wrote this:
<cite>
FRED is a useful concrete example. Under VMX, the FRED host state inIA32_FRED_CONFIG,IA32_FRED_STKLVLS, IA32_FRED_RSP1-3 and IA32_FRED_SSP1-3 is covered by the VMCS host-state area, so the TDX module is expected to restore these MSRs on TDH.VP.ENTER return.IA32_FRED_RSP0andIA32_PL0_SSP(a.k.a.IA32_FRED_SSP0) are handled by software, so the TDX module is expected to clobber them on TDH.VP.ENTER return. </cite>
Quoted 5 lines
That one reads as obviously right to me. If VMX and TDX differed in how
IA32_FRED_RSP0andIA32_PL0_SSPare handled, that would be a red flag.Did you go through all the MSRs and check that the VMX and TDX behavior matches today?
Not yet.
This topic is put in the cover letter for discussions.
And the patch series itself doesn't depend on conclusion of the discussions.
Quoted 2 lines
Thanks!
Edgecombe, Rick P
maintainerRe: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Binbin WuQuoted 6 lines · Binbin Wu
Quoted 5 lines
I think this actually surfaces another problem with TD-first enabling.
KVM_TDX_CAPABILITIESonly returns the directly configurable bits. Then recall,KVM_TDX_GET_CPUIDreturns the actual TDX module's view of CPUID bits to userspace. Then userspace callsKVM_SET_CPUIDto actually put them on KVM's vcpu so they can match between Qemu, KVM and TDX
Quoted 7 lines
That brings up a point..
Today, vcpu->arch.cpu_caps[] is capped by kvm_cpu_caps[] (plus a few special cases). As mentioned in the cover letter, this patch series doesn't enforce consistency between KVM's view and the guest's view of vCPU capabilities because KVM doesn't currently use its own view to make decisions for TDs (e.g. saving/restoring feature-related MSRs).
Not sure if I'm missing your point here. I don't think we ever want to have KVM enforce consistency between KVM's view and guests. We just need to provide enough info to userspace such that it can make them consistent.
Quoted 4 lines
However, if KVM starts making decisions for TDX based on vcpu-
Quoted 1 line
arch.cpu_caps[], intersecting userspace input with kvm_cpu_caps[] will not
work for TDX.
vcpu->arch.cpu_caps are actually already consulted for TDX. I remember seeing a bunch of the the guest cpuid feature checks during the base enabling, probably working on this problem. Let me what we have today.
From a Linux guest boot, guest_cpu_cap_has() returns true for:
xsave
smep
smap
fsgsbase
pku
la57
umip
vmx
pcid
lam
unknown
ibt
x2apic
Since we share code with normal VMs (and manage shared EPT in KVM), some checks are going to happen. If there is some new feature foo we enable for TDX. And later KVM adds new logic around vcpu->arch.cpu_caps for it, then there is a small risk of being pinned down when we want to add new guest_cpu_cap_has() logic for normal VMs. Since we already are hitting these checks for TDX, the general case is not theoretical.
Quoted 2 lines
I think this is probably needed in the future? If so, allowing features
outside of kvm_cpu_caps[] for TDX means
Yea, I think allowing TDX features outside of kvm_cpu_caps is for special cases. And filtering like you have is good.
Quoted 3 lines
we will need TDX-specific handling to construct KVM's view of vCPU
capabilities. That likely implies tracking all known/supported TDX features, which is doable, but it will make the allow list bigger.
In this thread we have been talking about what "normal VMs" support, but in the code and uAPI it really is about what KVM supports. If we let TDX use a feature that *KVM* doesn't support, it is the risky zone.
I say we punt on this. Let's remember it's dicey and if we find TDX feature enabling is being blocked all the time by normal VM enabling, we can work on a solution. Does anyone see any big risk of this being harder later than it is today?
I prefer to at least start filtering ASAP.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
Quoted 15 lines · Rick P
Quoted 6 lines · Binbin Wu
Quoted 5 lines
I think this actually surfaces another problem with TD-first enabling.
KVM_TDX_CAPABILITIESonly returns the directly configurable bits. Then recall,KVM_TDX_GET_CPUIDreturns the actual TDX module's view of CPUID bits to userspace. Then userspace callsKVM_SET_CPUIDto actually put them on KVM's vcpu so they can match between Qemu, KVM and TDXQuoted 7 lines
That brings up a point..
Today, vcpu->arch.cpu_caps[] is capped by kvm_cpu_caps[] (plus a few special cases). As mentioned in the cover letter, this patch series doesn't enforce consistency between KVM's view and the guest's view of vCPU capabilities because KVM doesn't currently use its own view to make decisions for TDs (e.g. saving/restoring feature-related MSRs).
Quoted 3 lines
Not sure if I'm missing your point here. I don't think we ever want to have KVM enforce consistency between KVM's view and guests. We just need to provide enough info to userspace such that it can make them consistent.
Consistency check prevents malicious userspace VMMs from lying to the KVM about some host state clobbering if KVM uses guest_cpu_cap_has() to management the state for TDX in the future.
Quoted 25 lines
Quoted 5 lines
However, if KVM starts making decisions for TDX based on vcpu-
Quoted 1 line
arch.cpu_caps[], intersecting userspace input with kvm_cpu_caps[] will not
work for TDX.
vcpu->arch.cpu_capsare actually already consulted for TDX. I remember seeing a bunch of the the guest cpuid feature checks during the base enabling, probably working on this problem. Let me what we have today.From a Linux guest boot,
guest_cpu_cap_has()returns true for:
xsave
smep
smap
fsgsbase
pku
la57
umip
vmx
pcid
lam
unknown
ibt
x2apic
Quoted 6 lines
Since we share code with normal VMs (and manage shared EPT in KVM), some checks are going to happen. If there is some new feature foo we enable for TDX. And later KVM adds new logic around
vcpu->arch.cpu_capsfor it, then there is a small risk of being pinned down when we want to add newguest_cpu_cap_has()logic for normal VMs. Since we already are hitting these checks for TDX, the general case is not theoretical.
... Yes, this was my concern.
If in the future KVM adds new guest_cpu_cap_has() for a host state clobbering feature and it is used TDX, it would cause problem if there is a mismatch between guest_cpu_cap_has() and the real value exposed to the TD.
Quoted 15 lines
Quoted 3 lines
I think this is probably needed in the future? If so, allowing features
outside of kvm_cpu_caps[] for TDX meansYea, I think allowing TDX features outside of
kvm_cpu_capsis for special cases. And filtering like you have is good.Quoted 3 lines
we will need TDX-specific handling to construct KVM's view of vCPU
capabilities. That likely implies tracking all known/supported TDX features, which is doable, but it will make the allow list bigger.In this thread we have been talking about what "normal VMs" support, but in the code and uAPI it really is about what KVM supports. If we let TDX use a feature that *KVM* doesn't support, it is the risky zone.
Quoted 6 lines
I say we punt on this. Let's remember it's dicey and if we find TDX feature enabling is being blocked all the time by normal VM enabling, we can work on a solution. Does anyone see any big risk of this being harder later than it is today?
I prefer to at least start filtering ASAP.
+1
Edgecombe, Rick P
maintainerRe: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
Quoted 11 lines · Binbin Wu
Quoted 6 lines
Since we share code with normal VMs (and manage shared EPT in KVM), some checks are going to happen. If there is some new feature foo we enable for TDX. And later KVM adds new logic around
vcpu->arch.cpu_capsfor it, then there is a small risk of being pinned down when we want to add newguest_cpu_cap_has()logic for normal VMs. Since we already are hitting these checks for TDX, the general case is not theoretical.... Yes, this was my concern.
If in the future KVM adds newguest_cpu_cap_has()for a host state clobbering feature and it is used TDX, it would cause problem if there is a mismatch betweenguest_cpu_cap_has()and the real value exposed to the TD.
We discussed offline that another benefit of filtering by kvm caps is that TDX will respect KVM's logic around disabling certain features, for example the enable_cet module param. Things like that could easily be added without consideration of TDX, and the reason for disabling them could possibly still apply to TDX. Or also surprise the user if TDX doesn't respect it. Better to be safe by default.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Binbin WuQuoted 7 lines · Binbin Wu
Quoted 6 lines · Rick P
Quoted 5 lines · Binbin Wu
Quoted 4 lines
I see. You added patch 2 because without it QEMU breaks. While for EST/TM2/SDBG/XTPR/DCA, QEMU doesn't break after they are turned to non-configurable. QEMU cannot represent all the userspace VMM. It still has the potential to breaks other userspace VMMs.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I think the risk is pretty low.
I am not sure if Sean could provide some insight about this in google's userspace VMM.Can't we fix the issue if we are wrong?
I think it could be fixed to add the missing bits (if any) as bug fix.
Directly configurable feature bits that KVM doesn't support for non-TDX VMs fall into five categories:
1) Features that are forced to 0 when #VE is reduced, or they lack KVM's support for associated MSRs:
EST, TM2, SDBG, DCA, ACPI, ACC (TM), RDT_A, RDT_M, TME, PCONFIG.
2) Features tied to MSR_IA32_MISC_ENABLE, whose corresponding IA32_MISC_ENABLE bit can't be set from a TD's point of view when TDCS.TD_CTLS.REDUCE_VE is set:
CID, PBE.
3) Host state clobbering features that KVM doesn't support for TDX:
HLE, RTM, WAITPKG, FRED.
Features in categories 1-3 should not be added to the allow list.
4) Features that are simply unsupported and thus don't need to be in the allow list:
PREFETCHWT1 (Xeon Phi only), PSN (not implemented by TDX-capable CPUs), AMX-TRANSPOSE (never implemented on any Intel platform), and RAO_INT (defined only for "future processors").
5) Features that are added to the allow list via TDX_CFG_EXTRA_F():
MWAIT, XTPR, HT, CORE_CAPABILITIES.
XTPR can be exposed to a TD, but it never takes effect in the underlying hardware when the guest changes IA32_MISC_ENABLE[23].
CORE_CAPABILITIES really belongs to category 1, but it's special because older specs defined it as a fixed-1 bit, so userspace VMMs may try to enable it based on stale knowledge. Add it to the allow list to accommodate the legacy TDX module definition. Reporting CORE_CAPABILITIES as directly configurable also lets userspace infer that the bit is no longer fixed-1, and correct its stale knowledge. Keep MSR_IA32_CORE_CAPS unsupported for TDX guests because existing TDX users have no guest access to MSR_IA32_CORE_CAPS.
With the features in category 5 in the allow list, it shouldn't break userspace.
If no objection, the next version will still limit the allow list to the capabilities KVM supported for non-TDX VMs, with a few exceptions listed in category 5.
Re: [PATCH v3 1/4] KVM: TDX: Track configurable CPUID bits allowed by KVM
In reply to Xiaoyao LiQuoted 29 lines · Xiaoyao Li
Quoted 20 lines · Rick P
Quoted 8 lines · Sean Christopherson
Yeah, *ideally* we'd magically enable everything everywhere all at once. In reality, different VM types are going to support features at different times. More importantly, as Xiaoyao points out below in #1, unless we enable everyting in a single patch, which is probably a terrible idea in most cases, we'll still end up with staged/progressive enabling, i.e. we still need to have patches that selectively enable and advertise a feature only for the VM types that actually support the feature.
Quoted 5 lines
This is all quite similar to Intel and AMD feature enabling being done at different times. The biggest difference is that Intel and AMD are mutually exclusive and so
KVM_GET_SUPPORTED_CPUIDalways reports the correct information, but TDX already providesKVM_TDX_CAPABILITIES, so AFAICT we still get accurate reporting for TDX, just in a slightly different way.I think this actually surfaces another problem with TD-first enabling.
KVM_TDX_CAPABILITIESonly returns the directly configurable bits. Then recall,KVM_TDX_GET_CPUIDreturns the actual TDX module's view of CPUID bits to userspace. Then userspace callsKVM_SET_CPUIDto actually put them on KVM's vcpu so they can match between Qemu, KVM and TDXQuoted 7 lines
So if a bit is enabled for
KVM_TDX_CAPABILITIES, but not yet inKVM_GET_SUPPORTED_CPUID. How should userspace interpretKVM_GET_SUPPORTED_CPUID? It can ignore it for TDX, but that is how it can find the PV bits today.> If we have a TD first feature, it could be a documentation update on how to interpret it. Or we could stuff the PV bits somewhere else for TDX and say to ignoreKVM_GET_SUPPORTED_CPUIDfor TDX. I think we don't need to solve it before we begin filtering like this series has.
Quoted 3 lines
The PV bits reported in
KVM_GET_SUPPORTED_CPUIDare the bits supported for non-TDX VMs. Only some of them are actually supported for TDX. I would suggest reporting TDX supported PV CPUIDs inKVM_TDX_CAPABILITIESas well.
I would leave it as a separate task if we want to report the PV features more accurately.
Currently, the CPUID bits reported by KVM_TDX_CAPABILITIES only serve the purpose for enumeration of directly configurable CPUID bits and the userspace VMM (e.g. QEMU) will follow the same structure and layout of the CPUIDs reported to construct the input for KVM_TDX_INIT_VM. If we report PV CPUIDs in KVM_TDX_CAPABILITIES, these values in KVM_TDX_INIT_VM will be rejected by the TDX module.