KVM: x86: Dynamically allocate hashed page list
Subsystems (4)
Sean Christopherson
maintainerv3 cover KVM: x86: Dynamically allocate hashed page list
Third times a charm! Right? Right!?!?
Allocate the hashed list of shadow pages dynamically (separate from struct kvm), and on-demand. The hashed list is 32KiB, i.e. absolutely belongs in a separate allocation, and is worth skipping if KVM isn't shadowing guest PTEs for the VM.
Side topic #1, a bunch of my measurements from v2 and ealier were "bad", because I was using a PROVE_LOCKING=y kernel, which significantly inflates the size of "struct kvm" in particular.
Side topic #2, I have a patch to dynamically allocate the memslots hash tables (they're very conveniently either 2KiB or 4KiB in size for 64-bit kernels), but I couldn't convince myself that the complexity is in any way justified. I did however account for the size of the hash tables in the assertions, if only to document where a big chunk of the per-VM memory usage is going.
Side topic #3, AFAIK, DEBUG_KERNEL=n builds are quite rare, so I'm banking on build bots tripping the assert (I'll also add a DEBUG_KERNEL=n config to my own testing, probably).
v3:
- Add comments explaining the {READ,WRITE}_ONCE logic, and why it's safe to set the list outside of mmu_lock. [Vipin]
- Make the assertions actually work. [Vipin]
- Refine the assertions so they (hopefully) won't fail on kernels with a bunch of debug crud added.
v2:
- https://lore.kernel.org/all/[email protected]
- Actually defer allocation when using TDP MMU. [Vipin]
- Free allocation on MMU teardown. [Vipin]
v1: https://lore.kernel.org/all/[email protected]
Sean Christopherson (3):
KVM: x86/mmu: Dynamically allocate shadow MMU's hashed page list
KVM: x86: Use kvzalloc() to allocate VM struct
KVM: x86/mmu: Defer allocation of shadow MMU's hashed page list
arch/x86/include/asm/kvm_host.h | 6 +-- arch/x86/kvm/mmu/mmu.c | 73 ++++++++++++++++++++++++++++++--- arch/x86/kvm/svm/svm.c | 2 + arch/x86/kvm/vmx/main.c | 2 + arch/x86/kvm/vmx/vmx.c | 2 + arch/x86/kvm/x86.c | 5 ++- arch/x86/kvm/x86.h | 22 ++++++++++ 7 files changed, 102 insertions(+), 10 deletions(-)
base-commit: 7ef51a41466bc846ad794d505e2e34ff97157f7f
Signature · 1 line
2.49.0.1112.g889b7c5bd8-goog
Sean Christopherson
maintainerPATCH v3 1/3 KVM: x86/mmu: Dynamically allocate shadow MMU's hashed page list
Dynamically allocate the (massive) array of hashed lists used to track shadow pages, as the array itself is 32KiB, i.e. is an order-3 allocation all on its own, and is *exactly* an order-3 allocation. Dynamically allocating the array will allow allocating "struct kvm" using kvmalloc(), and will also allow deferring allocation of the array until it's actually needed, i.e. until the first shadow root is allocated.
Opportunistically use kvmalloc() for the hashed lists, as an order-3 allocation is (stating the obvious) less likely to fail than an order-4 allocation, and the overhead of vmalloc() is undesirable given that the size of the allocation is fixed.
--- arch/x86/include/asm/kvm_host.h | 4 ++-- arch/x86/kvm/mmu/mmu.c | 23 ++++++++++++++++++++++- arch/x86/kvm/x86.c | 5 ++++- 3 files changed, 28 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -2
@@ -1343,7 +1343,7 @@ struct kvm_arch {bool has_private_mem;bool has_protected_state;bool pre_fault_allowed; -struct hlist_head mmu_page_hash[KVM_NUM_MMU_PAGES ]; +struct hlist_head *mmu_page_hash;struct list_head active_mmu_pages;/* * A list of kvm_mmu_page structs that, if zapped, could possibly be
@@ -2006,7 +2006,7 @@ void kvm_mmu_vendor_module_exit(void);void kvm_mmu_destroy (struct kvm_vcpu *vcpu);int kvm_mmu_create (struct kvm_vcpu *vcpu); -void kvm_mmu_init_vm (struct kvm *kvm); +int kvm_mmu_init_vm (struct kvm *kvm);void kvm_mmu_uninit_vm (struct kvm *kvm); void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
arch/x86/kvm/mmu/mmu.c +22 -1
@@ -3882,6 +3882,18 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)return r; } +static int kvm_mmu_alloc_page_hash (struct kvm *kvm) +{ + typeof(kvm->arch.mmu_page_hash) h; + + h =kvcalloc (KVM_NUM_MMU_PAGES ,sizeof (*h),GFP_KERNEL_ACCOUNT ); +if (!h) +return -ENOMEM ; + + kvm->arch.mmu_page_hash = h; +return 0 ; +} +static int mmu_first_shadow_root_alloc (struct kvm *kvm) {struct kvm_memslots *slots;
@@ -6675,13 +6687,19 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)kvm_tdp_mmu_zap_invalidated_roots (kvm, true); } -void kvm_mmu_init_vm (struct kvm *kvm) +int kvm_mmu_init_vm (struct kvm *kvm) { +int r; + kvm->arch.shadow_mmio_value = shadow_mmio_value;INIT_LIST_HEAD (&kvm->arch.active_mmu_pages);INIT_LIST_HEAD (&kvm->arch.possible_nx_huge_pages);spin_lock_init (&kvm->arch.mmu_unsync_pages_lock); + r =kvm_mmu_alloc_page_hash (kvm); +if (r) +return r; +if (tdp_mmu_enabled)kvm_mmu_init_tdp_mmu (kvm);
@@ -6692,6 +6710,7 @@ void kvm_mmu_init_vm(struct kvm *kvm) kvm->arch.split_desc_cache.kmem_cache = pte_list_desc_cache; kvm->arch.split_desc_cache.gfp_zero = __GFP_ZERO; +return 0 ; }static void mmu_free_vm_memory_caches (struct kvm *kvm)
@@ -6703,6 +6722,8 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)void kvm_mmu_uninit_vm (struct kvm *kvm) { +kvfree (kvm->arch.mmu_page_hash); +if (tdp_mmu_enabled)kvm_mmu_uninit_tdp_mmu (kvm);
arch/x86/kvm/x86.c +4 -1
@@ -12787,7 +12787,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)if (ret) goto out; -kvm_mmu_init_vm (kvm); + ret =kvm_mmu_init_vm (kvm); +if (ret) + goto out_cleanup_page_track; ret =kvm_x86_call (vm_init)(kvm);if (ret)
@@ -12840,6 +12842,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) out_uninit_mmu:kvm_mmu_uninit_vm (kvm); +out_cleanup_page_track:kvm_page_track_cleanup (kvm); out:return ret;
Signature · 1 line
2.49.0.1112.g889b7c5bd8-goog
Sean Christopherson
maintainerPATCH v3 2/3 KVM: x86: Use kvzalloc() to allocate VM struct
In reply to Sean ChristophersonAllocate VM structs via kvzalloc(), i.e. try to use a contiguous physical allocation before falling back to __vmalloc(), to avoid the overhead of establishing the virtual mappings. For non-debug builds, The SVM and VMX (and TDX) structures are now just below 7000 bytes in the worst case scenario (see below), i.e. are order-1 allocations, and will likely remain that way for quite some time.
Add compile-time assertions in vendor code to ensure the size of the structures, sans the memslos hash tables, are order-0 allocations, i.e. are less than 4KiB. There's nothing fundamentally wrong with a larger kvm_{svm,vmx,tdx} size, but given that the size of the structure (without the memslots hash tables) is below 2KiB after 18+ years of existence, more than doubling the size would be quite notable.
Add sanity checks on the memslot hash table sizes, partly to ensure they aren't resized without accounting for the impact on VM structure size, and partly to document that the majority of the size of VM structures comes from the memslots.
--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/vmx/main.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 2 ++ arch/x86/kvm/x86.h | 22 ++++++++++++++++++++++ 5 files changed, 29 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +1 -1
@@ -1961,7 +1961,7 @@ void kvm_x86_vendor_exit(void);#define __KVM_HAVE_ARCH_VM_ALLOCstatic inline struct kvm *kvm_arch_alloc_vm (void ) { -return __vmalloc (kvm_x86_ops.vm_size,GFP_KERNEL_ACCOUNT | __GFP_ZERO); +return kvzalloc (kvm_x86_ops.vm_size,GFP_KERNEL_ACCOUNT ); }#define __KVM_HAVE_ARCH_VM_FREE
arch/x86/kvm/svm/svm.c +2 -0
@@ -5675,6 +5675,8 @@ static int __init svm_init(void) {int r; +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_svm); +__unused_size_checks ();if (!kvm_is_svm_supported ())
arch/x86/kvm/vmx/main.c +2 -0
@@ -64,6 +64,8 @@ static __init int vt_hardware_setup(void) vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; } +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_tdx); +return 0 ; }
arch/x86/kvm/vmx/vmx.c +2 -0
@@ -8643,6 +8643,8 @@ int __init vmx_init(void) {int r, cpu; +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_vmx); +if (!kvm_is_vmx_supported ())return -EOPNOTSUPP ;
arch/x86/kvm/x86.h +22 -0
@@ -55,6 +55,28 @@ struct kvm_host_values {void kvm_spurious_fault (void ); +#define SIZE_OF_MEMSLOTS_HASHTABLE \ + (sizeof(((struct kvm_memslots *)0)->id_hash) * 2 * KVM_MAX_NR_ADDRESS_SPACES) + +/* Sanity check the size of the memslot hash tables. */ +static_assert (SIZE_OF_MEMSLOTS_HASHTABLE == + (1024 * (1 +IS_ENABLED (CONFIG_X86_64 )) * (1 +IS_ENABLED (CONFIG_KVM_SMM )))); + +/* +* Assert that "struct kvm_{svm,vmx,tdx}" is an order-0 or order-1 allocation. +* Spilling over to an order-2 allocation isn't fundamentally problematic, but +* isn't expected to happen in the foreseeable future (O(years)). Assert that +* the size is an order-0 allocation when ignoring the memslot hash tables, to +* help detect and debug unexpected size increases. +*/ +#define KVM_SANITY_CHECK_VM_STRUCT_SIZE (x) \ +do { \ + BUILD_BUG_ON(get_order(sizeof(struct x) - SIZE_OF_MEMSLOTS_HASHTABLE) && \ + !IS_ENABLED(CONFIG_DEBUG_KERNEL) && !IS_ENABLED(CONFIG_KASAN)); \ + BUILD_BUG_ON(get_order(sizeof(struct x)) < 2 && \ + !IS_ENABLED(CONFIG_DEBUG_KERNEL) && !IS_ENABLED(CONFIG_KASAN)); \ +} while (0) +#define KVM_NESTED_VMENTER_CONSISTENCY_CHECK (consistency_check) \ ({ \ bool failed = (consistency_check); \
Signature · 1 line
2.49.0.1112.g889b7c5bd8-goog
Paolo Bonzini
maintainerRe: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 59 lines · Sean Christopherson
Allocate VM structs via
kvzalloc(), i.e. try to use a contiguous physical allocation before falling back to__vmalloc(), to avoid the overhead of establishing the virtual mappings. For non-debug builds, The SVM and VMX (and TDX) structures are now just below 7000 bytes in the worst case scenario (see below), i.e. are order-1 allocations, and will likely remain that way for quite some time.Add compile-time assertions in vendor code to ensure the size of the structures, sans the memslos hash tables, are order-0 allocations, i.e. are less than 4KiB. There's nothing fundamentally wrong with a larger kvm_{svm,vmx,tdx} size, but given that the size of the structure (without the memslots hash tables) is below 2KiB after 18+ years of existence, more than doubling the size would be quite notable.
Add sanity checks on the memslot hash table sizes, partly to ensure they aren't resized without accounting for the impact on VM structure size, and partly to document that the majority of the size of VM structures comes from the memslots.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/vmx/main.c | 2 ++ arch/x86/kvm/vmx/vmx.c | 2 ++ arch/x86/kvm/x86.h | 22 ++++++++++++++++++++++ 5 files changed, 29 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +1
-1@@ -1961,7 +1961,7 @@ void kvm_x86_vendor_exit(void);#define __KVM_HAVE_ARCH_VM_ALLOCstatic inline struct kvm *kvm_arch_alloc_vm (void ) { -return __vmalloc (kvm_x86_ops.vm_size,GFP_KERNEL_ACCOUNT | __GFP_ZERO); +return kvzalloc (kvm_x86_ops.vm_size,GFP_KERNEL_ACCOUNT ); }#define __KVM_HAVE_ARCH_VM_FREEarch/x86/kvm/svm/svm.c +2
-0@@ -5675,6 +5675,8 @@ static int __init svm_init(void) {int r; +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_svm); +__unused_size_checks ();if (!kvm_is_svm_supported ())arch/x86/kvm/vmx/main.c +0
-0
arch/x86/kvm/vmx/main.c static __init int vt_hardware_setup(void) · line 67
Quoted 4 lines
arch/x86/kvm/vmx/main.c +1
-0@@ -64,6 +64,8 @@ static __init int vt_hardware_setup(void) vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; } +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_tdx);
I would put either both or no checks in main.c.
Or if you use static_assert, you can also place the macro invocation close to the struct definition.
Paolo
Quoted 11 lines
arch/x86/kvm/vmx/main.c +8
-0+*/ +#define KVM_SANITY_CHECK_VM_STRUCT_SIZE (x) \ +do { \ + BUILD_BUG_ON(get_order(sizeof(struct x) - SIZE_OF_MEMSLOTS_HASHTABLE) && \ + !IS_ENABLED(CONFIG_DEBUG_KERNEL) && !IS_ENABLED(CONFIG_KASAN)); \ + BUILD_BUG_ON(get_order(sizeof(struct x)) < 2 && \ + !IS_ENABLED(CONFIG_DEBUG_KERNEL) && !IS_ENABLED(CONFIG_KASAN)); \ +} while (0)#define KVM_NESTED_VMENTER_CONSISTENCY_CHECK (consistency_check) \ ({ \ bool failed = (consistency_check); \
Sean Christopherson
maintainerRe: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 14 lines · Paolo Bonzini
Quoted 13 lines · Sean Christopherson
arch/x86/kvm/svm/svm.c +2
-0@@ -5675,6 +5675,8 @@ static int __init svm_init(void) {int r; +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_svm); +__unused_size_checks ();if (!kvm_is_svm_supported ())arch/x86/kvm/vmx/main.c +0
-0
Quoted 8 lines
arch/x86/kvm/vmx/main.c static __init int vt_hardware_setup(void) · line 66
Quoted 6 lines
arch/x86/kvm/vmx/main.c +1
-0@@ -64,6 +64,8 @@ static __init int vt_hardware_setup(void) vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; } +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_tdx);I would put either both or no checks in
main.c.
Yeah, I agree the current split is ugly. I originally had 'em both in main.c, but then the assert effectively becomes dependent on CONFIG_KVM_INTEL_TDX=y.
Aha! If we add a proper tdx_hardware_setup(), then there's a convenient location for the assert, IMO it's much easier to see/document the "TDX module not loaded" behavior, and the TDX-specific kvm_x86_ops hooks don't need to be visible symbols.
I'll slot the below in, unless you've got a better idea.
Quoted 2 lines
Or if you use
static_assert, you can also place the macro invocation close to the struct definition.
Already tried that, get_order() doesn't play nice with static_assert :-/
--
From: Sean Christopherson <[email protected]>
Date: Mon, 19 May 2025 07:15:27 -0700
Subject: [PATCH] KVM: TDX: Move TDX hardware setup from main.c to tdx.c
Move TDX hardware setup to tdx.c, as the code is obviously TDX specific, co-locating the setup with tdx_bringup() makes it easier to see and document the success_disable_tdx "error" path, and configuring the TDX specific hooks in tdx.c reduces the number of globally visible TDX symbols.
--- arch/x86/kvm/vmx/main.c | 36 ++--------------------------- arch/x86/kvm/vmx/tdx.c | 46 +++++++++++++++++++++++++++----------- arch/x86/kvm/vmx/tdx.h | 1 + arch/x86/kvm/vmx/x86_ops.h | 10 --------- 4 files changed, 36 insertions(+), 57 deletions(-)
arch/x86/kvm/vmx/main.c +2 -34
@@ -29,40 +29,8 @@ static __init int vt_hardware_setup(void)if (ret)return ret; -/* -* Update vt_x86_ops::vm_size here so it is ready before -* kvm_ops_update() is called in kvm_x86_vendor_init(). -* -* Note, the actual bringing up of TDX must be done after -* kvm_ops_update() because enabling TDX requires enabling -* hardware virtualization first, i.e., all online CPUs must -* be in post-VMXON state. This means the @vm_size here -* may be updated to TDX's size but TDX may fail to enable -* at later time. -* -* The VMX/VT code could update kvm_x86_ops::vm_size again -* after bringing up TDX, but this would require exporting -* either kvm_x86_ops or kvm_ops_update() from the base KVM -* module, which looks overkill. Anyway, the worst case here -* is KVM may allocate couple of more bytes than needed for -* each VM. -*/ -if (enable_tdx) { - vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size, -sizeof (struct kvm_tdx )); -/* -* Note, TDX may fail to initialize in a later time in -* vt_init(), in which case it is not necessary to setup -* those callbacks. But making them valid here even -* when TDX fails to init later is fine because those -* callbacks won't be called if the VM isn't TDX guest. -*/ - vt_x86_ops.link_external_spt = tdx_sept_link_private_spt; - vt_x86_ops.set_external_spte = tdx_sept_set_private_spte; - vt_x86_ops.free_external_spt = tdx_sept_free_private_spt; - vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte; - vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; - } +if (enable_tdx) +tdx_hardware_setup ();return 0 ; }
arch/x86/kvm/vmx/tdx.c +33 -13
85 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +1 -0
@@ -8,6 +8,7 @@#ifdef CONFIG_KVM_INTEL_TDX #include "common.h" +void tdx_hardware_setup (void );int tdx_bringup (void );void tdx_cleanup (void );
arch/x86/kvm/vmx/x86_ops.h +0 -10
@@ -136,7 +136,6 @@ int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu);fastpath_t tdx_vcpu_run (struct kvm_vcpu *vcpu,bool force_immediate_exit);void tdx_prepare_switch_to_guest (struct kvm_vcpu *vcpu);void tdx_vcpu_put (struct kvm_vcpu *vcpu); -bool tdx_protected_apic_has_interrupt (struct kvm_vcpu *vcpu);int tdx_handle_exit (struct kvm_vcpu *vcpu,enum exit_fastpath_completion fastpath);
@@ -151,15 +150,6 @@ int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr);int tdx_vcpu_ioctl (struct kvm_vcpu *vcpu,void __user *argp); -int tdx_sept_link_private_spt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,void *private_spt); -int tdx_sept_free_private_spt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,void *private_spt); -int tdx_sept_set_private_spte (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn); -int tdx_sept_remove_private_spte (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn); -void tdx_flush_tlb_current (struct kvm_vcpu *vcpu);void tdx_flush_tlb_all (struct kvm_vcpu *vcpu);void tdx_load_mmu_pgd (struct kvm_vcpu *vcpu,hpa_t root_hpa,int root_level);
base-commit: 7ef51a41466bc846ad794d505e2e34ff97157f7f --
Paolo Bonzini
maintainerRe: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 1 line · Sean Christopherson
I'll slot the below in, unless you've got a better idea.
Nope, that's a good idea. Should have thought about it a couple months ago.
Paolo
Re: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
In reply to Sean ChristophersonQuoted 24 lines · Sean Christopherson
Quoted 14 lines · Paolo Bonzini
Quoted 13 lines · Sean Christopherson
arch/x86/kvm/svm/svm.c +2
-0@@ -5675,6 +5675,8 @@ static int __init svm_init(void) {int r; +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_svm); +__unused_size_checks ();if (!kvm_is_svm_supported ())arch/x86/kvm/vmx/main.c +0
-0Quoted 8 lines
arch/x86/kvm/vmx/main.c static __init int vt_hardware_setup(void) · line 66
Quoted 6 lines
arch/x86/kvm/vmx/main.c +1
-0@@ -64,6 +64,8 @@ static __init int vt_hardware_setup(void) vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; } +KVM_SANITY_CHECK_VM_STRUCT_SIZE (kvm_tdx);I would put either both or no checks in
main.c.
Quoted 8 lines
Yeah, I agree the current split is ugly. I originally had 'em both in
main.c, but then the assert effectively becomes dependent onCONFIG_KVM_INTEL_TDX=y.Aha! If we add a proper
tdx_hardware_setup(), then there's a convenient location for the assert, IMO it's much easier to see/document the "TDX module not loaded" behavior, and the TDX-specifickvm_x86_opshooks don't need to be visible symbols.I'll slot the below in, unless you've got a better idea.
Looks good to me too. Minor things below.
[...]
Quoted 26 lines
(quoted diff) +10
-4+static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) { struct page *page = pfn_to_page(pfn); int ret;@@ -3507,10 +3507,14 @@ int __init tdx_bringup(void) r = __tdx_bringup(); if (r) { /* - * Disable TDX only but don't fail to load module if - * the TDX module could not be loaded. No need to print - * message saying "module is not loaded" because it was - * printed when the first SEAMCALL failed. + * Disable TDX only but don't fail to load module if the TDX + * module could not be loaded. No need to print message saying + * "module is not loaded" because it was printed when the first + * SEAMCALL failed. Don't bother unwinding the S-EPT hooks or + * vm_size, as kvm_x86_ops have already been finalized (and are + * intentionally not exported). The S-EPT code is unreachable, + * and allocating a few more bytes per VM in a should-be-rare + * failure scenario is a non-issue. */ if (r == -ENODEV) goto success_disable_tdx;@@ -3524,3 +3528,19 @@ int __init tdx_bringup(void) enable_tdx = 0;
Review int __init tdx_bringup(void) · line 3538
Quoted 10 lines
(quoted diff) +8
-0return 0; } + + +void __init tdx_hardware_setup(void) +{ + /* + * Note, if the TDX module can't be loaded, KVM TDX support will be + * disabled but KVM will continue loading (see tdx_bringup()). + */
This comment seems a little bit weird to me. I think what you meant here is the @vm_size and those S-EPT ops are not unwound while TDX cannot be brought up but KVM is still loaded.
Quoted 13 lines
(quoted diff) +8
-0+ vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size, sizeof(struct kvm_tdx)); + + vt_x86_ops.link_external_spt = tdx_sept_link_private_spt; + vt_x86_ops.set_external_spte = tdx_sept_set_private_spte; + vt_x86_ops.free_external_spt = tdx_sept_free_private_spt; + vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte; + vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; +}arch/x86/kvm/vmx/tdx.h +0
-0
arch/x86/kvm/vmx/tdx.h line 14
Quoted 7 lines
arch/x86/kvm/vmx/tdx.h +1
-0@@ -8,6 +8,7 @@#ifdef CONFIG_KVM_INTEL_TDX #include "common.h" +void tdx_hardware_setup (void );int tdx_bringup (void );void tdx_cleanup (void );
There's a build error when CONFIG_KVM_INTEL_TDX is off:
vmx/main.c: In function ‘vt_hardware_setup’:vmx/main.c:34:17: error: implicit declaration of function ‘tdx_hardware_setup’; did you mean ‘vmx_hardware_setup’? [-Wimplicit-function-declaration]
34 | tdx_hardware_setup();
| ^~~~~~~~~~~~~~~~~~
| vmx_hardware_setup.. for which you need a stub for tdx_hardware_setup() when CONFIG_KVM_INTEL_TDX is off.
And one more thing:
With the above patch, we still have below code in vt_init():
/*
* TDX and VMX have different vCPU structures. Calculate the
* maximum size/align so that kvm_init() can use the larger
* values to create the kmem_vcpu_cache.
*/
vcpu_size = sizeof(struct vcpu_vmx);
vcpu_align = __alignof__(struct vcpu_vmx);
if (enable_tdx) {
vcpu_size = max_t(unsigned, vcpu_size,
sizeof(struct vcpu_tdx));
vcpu_align = max_t(unsigned, vcpu_align,
__alignof__(struct vcpu_tdx));
kvm_caps.supported_vm_types |= BIT(KVM_X86_TDX_VM);
}It's kinda ugly too IMHO.
Since we already have @vm_size in kvm_x86_ops, how about also adding vcpu_size and vcpu_align to it? Then they can be treated in the same way as vm_size for TDX.
They are not needed for SVM, but it doesn't hurt that much?
Compile test only:
From b3a4b29cbd371189860b103813354813912019fa Mon Sep 17 00:00:00 2001
From: Kai Huang <[email protected]>
Date: Wed, 21 May 2025 10:32:28 +1200
Subject: [PATCH] vcpu size/align for TDX
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/svm/svm.c | 6 ++++-- arch/x86/kvm/vmx/main.c | 20 +++----------------- arch/x86/kvm/vmx/tdx.c | 7 +++++++ 4 files changed, 16 insertions(+), 19 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1684,6 +1684,8 @@ struct kvm_x86_ops {void (*vcpu_after_set_cpuid)(struct kvm_vcpu *vcpu);unsigned int vm_size; +unsigned int vcpu_size; +unsigned int vcpu_align;int (*vm_init)(struct kvm *kvm);void (*vm_destroy)(struct kvm *kvm);void (*vm_pre_destroy)(struct kvm *kvm);
arch/x86/kvm/svm/svm.c +4 -2
@@ -5094,6 +5094,9 @@ static struct kvm_x86_ops svm_x86_ops __initdata = { .vcpu_reset = svm_vcpu_reset, .vm_size = sizeof(struct kvm_svm), + .vcpu_size = sizeof(struct vcpu_svm), + .vcpu_align = __alignof__(struct vcpu_svm), + .vm_init = svm_vm_init, .vm_destroy = svm_vm_destroy,
@@ -5543,8 +5546,7 @@ static int __init svm_init(void)* Common KVM initialization _must_ come last, after this, /dev/kvm is * exposed to userspace! */ - r =kvm_init (sizeof (struct vcpu_svm ), __alignof__(struct vcpu_svm ), -THIS_MODULE ); + r = kvm_init(svm_x86_ops.vcpu_size, svm_x86_ops.vcpu_align,
THIS_MODULE);
if (r)
goto err_kvm_init;arch/x86/kvm/vmx/main.c +3 -17
@@ -879,6 +879,8 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .has_emulated_msr = vt_has_emulated_msr, .vm_size = sizeof(struct kvm_vmx), + .vcpu_size = sizeof(struct vcpu_vmx), + .vcpu_align = __alignof__(struct vcpu_vmx), .vm_init = vt_vm_init, .vm_pre_destroy = vt_vm_pre_destroy,
@@ -1035,7 +1037,6 @@ module_exit(vt_exit);static int __init vt_init (void ) { -unsigned vcpu_size, vcpu_align;int r; r =vmx_init ();
@@ -1047,26 +1048,11 @@ static int __init vt_init(void)if (r) goto err_tdx_bringup; -/* -* TDX and VMX have different vCPU structures. Calculate the -* maximum size/align so that kvm_init() can use the larger -* values to create the kmem_vcpu_cache. -*/ - vcpu_size =sizeof (struct vcpu_vmx ); - vcpu_align = __alignof__(struct vcpu_vmx ); -if (enable_tdx) { - vcpu_size =max_t (unsigned, vcpu_size, -sizeof (struct vcpu_tdx )); - vcpu_align =max_t (unsigned, vcpu_align, - __alignof__(struct vcpu_tdx )); - kvm_caps.supported_vm_types |=BIT (KVM_X86_TDX_VM ); - } -/* * Common KVM initialization _must_ come last, after this, /dev/kvm is * exposed to userspace! */ - r =kvm_init (vcpu_size, vcpu_align,THIS_MODULE ); + r =kvm_init (vt_x86_ops.vcpu_size, vt_x86_ops.vcpu_align,THIS_MODULE );if (r) goto err_kvm_init;
arch/x86/kvm/vmx/tdx.c +3 -0
@@ -3522,6 +3522,9 @@ int __init tdx_bringup(void) enable_tdx =0 ; } +if (enable_tdx) + kvm_caps.supported_vm_types |=BIT (KVM_X86_TDX_VM ); +return r; success_disable_tdx:
@@ -3537,6 +3540,10 @@ void __init tdx_hardware_setup(void)* disabled but KVM will continue loading (see tdx_bringup()). */ vt_x86_ops.vm_size = max_t(unsigned int, vt_x86_ops.vm_size,
sizeof(struct kvm_tdx));
+ vt_x86_ops.vcpu_size = max_t(unsigned int, vt_x86_ops.vcpu_size,
+ sizeof(struct vcpu_tdx));
+ vt_x86_ops.vcpu_align = max_t(unsigned int, vt_x86_ops.vcpu_align,
+ __alignof__(struct vcpu_tdx));
vt_x86_ops.link_external_spt = tdx_sept_link_private_spt;
vt_x86_ops.set_external_spte = tdx_sept_set_private_spte;Signature · 1 line
2.49.0
Sean Christopherson
maintainerRe: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 33 lines · Kai Huang
Quoted 32 lines · Sean Christopherson
(quoted diff) +14
-4+static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) { struct page *page = pfn_to_page(pfn); int ret;@@ -3507,10 +3507,14 @@ int __init tdx_bringup(void) r = __tdx_bringup(); if (r) { /* - * Disable TDX only but don't fail to load module if - * the TDX module could not be loaded. No need to print - * message saying "module is not loaded" because it was - * printed when the first SEAMCALL failed. + * Disable TDX only but don't fail to load module if the TDX + * module could not be loaded. No need to print message saying + * "module is not loaded" because it was printed when the first + * SEAMCALL failed. Don't bother unwinding the S-EPT hooks or + * vm_size, as kvm_x86_ops have already been finalized (and are + * intentionally not exported). The S-EPT code is unreachable, + * and allocating a few more bytes per VM in a should-be-rare + * failure scenario is a non-issue. */ if (r == -ENODEV) goto success_disable_tdx;@@ -3524,3 +3528,19 @@ int __init tdx_bringup(void) enable_tdx = 0; return 0; } + + +void __init tdx_hardware_setup(void) +{
Quoted 8 lines
Review int __init tdx_bringup(void) · line 3538
Quoted 4 lines
(quoted diff) +4
-0+ /* + * Note, if the TDX module can't be loaded, KVM TDX support will be + * disabled but KVM will continue loading (see tdx_bringup()). + */This comment seems a little bit weird to me. I think what you meant here is the @vm_size and those S-EPT ops are not unwound while TDX cannot be brought up but KVM is still loaded.
This comment is weird? Or the one in tdx_bringup() is weird? The sole intent of _this_ comment is to clarify that KVM could still end up running load with TDX disabled. The comment about not unwinding S-EPT resides in tdx_bringup(), because that's where the actual decision to not reject KVM load and to not undo the setup lives.
Quoted 25 lines
Quoted 12 lines
(quoted diff) +7
-0+ + vt_x86_ops.link_external_spt = tdx_sept_link_private_spt; + vt_x86_ops.set_external_spte = tdx_sept_set_private_spte; + vt_x86_ops.free_external_spt = tdx_sept_free_private_spt; + vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte; + vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; +}arch/x86/kvm/vmx/tdx.h +0
-0arch/x86/kvm/vmx/tdx.h line 14
Quoted 7 lines
arch/x86/kvm/vmx/tdx.h +1
-0@@ -8,6 +8,7 @@#ifdef CONFIG_KVM_INTEL_TDX #include "common.h" +void tdx_hardware_setup (void );int tdx_bringup (void );void tdx_cleanup (void );There's a build error when
CONFIG_KVM_INTEL_TDXis off:
vmx/main.c: In function ‘vt_hardware_setup’:vmx/main.c:34:17: error: implicit declaration of function ‘tdx_hardware_setup’; did you mean ‘vmx_hardware_setup’? [-Wimplicit-function-declaration]
Quoted 6 lines
34 | tdx_hardware_setup(); | ^~~~~~~~~~~~~~~~~~ | vmx_hardware_setup.. for which you need a stub for
tdx_hardware_setup()whenCONFIG_KVM_INTEL_TDXis off.
Not in kvm-x86/next, commit 907092bf7cbd ("KVM: VMX: Clean up and macrofy x86_ops") buried all of vt_hardware_setup() behind CONFIG_KVM_INTEL_TDX=y.
Quoted 19 lines
And one more thing:
With the above patch, we still have below code in
vt_init():/* * TDX and VMX have different vCPU structures. Calculate the * maximum size/align so that kvm_init() can use the larger * values to create the kmem_vcpu_cache. */ vcpu_size = sizeof(struct vcpu_vmx); vcpu_align = __alignof__(struct vcpu_vmx); if (enable_tdx) { vcpu_size = max_t(unsigned, vcpu_size, sizeof(struct vcpu_tdx)); vcpu_align = max_t(unsigned, vcpu_align, __alignof__(struct vcpu_tdx)); kvm_caps.supported_vm_types |= BIT(KVM_X86_TDX_VM); }
Quoted 7 lines
It's kinda ugly too IMHO.
Since we already have @vm_size in
kvm_x86_ops, how about also addingvcpu_sizeandvcpu_alignto it? Then they can be treated in the same way asvm_sizefor TDX.They are not needed for SVM, but it doesn't hurt that much?
I'd rather not. vt_init() already needs to be aware of TDX, e.g. to call into tdx_bringup() in the first place. Shoving state into kvm_x86_ops that is only ever used in vt_init() (an __init function at that) isn't a net positive.
Putting the fields in kvm_x86_init_ops would be better, but I still don't think the complexity and indirection is justified. Bleeding gory TDX details into the common code is undesirable, but I don't see the size of vcpu_tdx or the fact that enable_tdx==true means KVM_X86_TDX_VM is supported as being information with hiding.
kvm_x86_ops.vm_size is a means to an end. E.g. if kvm_vcpu_cache didn't exist, then we'd probably want/need kvm_x86_ops.vcpu_size, but it does exist, so...
Re: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 43 lines · Sean Christopherson
Quoted 33 lines · Kai Huang
Quoted 32 lines · Sean Christopherson
(quoted diff) +14
-4+static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) { struct page *page = pfn_to_page(pfn); int ret;@@ -3507,10 +3507,14 @@ int __init tdx_bringup(void) r = __tdx_bringup(); if (r) { /* - * Disable TDX only but don't fail to load module if - * the TDX module could not be loaded. No need to print - * message saying "module is not loaded" because it was - * printed when the first SEAMCALL failed. + * Disable TDX only but don't fail to load module if the TDX + * module could not be loaded. No need to print message saying + * "module is not loaded" because it was printed when the first + * SEAMCALL failed. Don't bother unwinding the S-EPT hooks or + * vm_size, as kvm_x86_ops have already been finalized (and are + * intentionally not exported). The S-EPT code is unreachable, + * and allocating a few more bytes per VM in a should-be-rare + * failure scenario is a non-issue. */ if (r == -ENODEV) goto success_disable_tdx;@@ -3524,3 +3528,19 @@ int __init tdx_bringup(void) enable_tdx = 0; return 0; } + + +void __init tdx_hardware_setup(void) +{Quoted 8 lines
Review int __init tdx_bringup(void) · line 3538
Quoted 4 lines
(quoted diff) +4
-0+ /* + * Note, if the TDX module can't be loaded, KVM TDX support will be + * disabled but KVM will continue loading (see tdx_bringup()). + */This comment seems a little bit weird to me. I think what you meant here is the @vm_size and those S-EPT ops are not unwound while TDX cannot be brought up but KVM is still loaded.
Quoted 2 lines
This comment is weird? Or the one in
tdx_bringup()is weird?
I definitely agree tdx_bringup() is weird :-)
Quoted 4 lines
The sole intent
of _this_ comment is to clarify that KVM could still end up running load with TDX disabled.
But this behaviour itself doesn't mean anything, e.g., if we export kvm_x86_ops, we could unwind them. So without mentioning "those are not unwound", it doesn't seem useful to me.
But it does have "(see tdx_bringup())" at the end, so OK to me. I guess I just wish it could be more verbose.
Quoted 3 lines
The comment about not unwinding S-EPT resides in
tdx_bringup(), because that's where the actual decision to not reject KVM load and to not undo the setup lives.
Right.
Quoted 30 lines
Quoted 30 lines
Quoted 13 lines
(quoted diff) +7
-0+ + vt_x86_ops.link_external_spt = tdx_sept_link_private_spt; + vt_x86_ops.set_external_spte = tdx_sept_set_private_spte; + vt_x86_ops.free_external_spt = tdx_sept_free_private_spt; + vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte; + vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt; +}arch/x86/kvm/vmx/tdx.h +0
-0arch/x86/kvm/vmx/tdx.h line 14
Quoted 7 lines
arch/x86/kvm/vmx/tdx.h +1
-0@@ -8,6 +8,7 @@#ifdef CONFIG_KVM_INTEL_TDX #include "common.h" +void tdx_hardware_setup (void );int tdx_bringup (void );void tdx_cleanup (void );There's a build error when
CONFIG_KVM_INTEL_TDXis off:
vmx/main.c: In function ‘vt_hardware_setup’:vmx/main.c:34:17: error: implicit declaration of function ‘tdx_hardware_setup’; did you mean ‘vmx_hardware_setup’? [-Wimplicit-function-declaration]34 | tdx_hardware_setup(); | ^~~~~~~~~~~~~~~~~~ | vmx_hardware_setup
Quoted 5 lines
Quoted 2 lines
.. for which you need a stub for
tdx_hardware_setup()whenCONFIG_KVM_INTEL_TDXis off.Not in kvm-x86/next, commit
907092bf7cbd("KVM: VMX: Clean up and macrofyx86_ops") buried all ofvt_hardware_setup()behindCONFIG_KVM_INTEL_TDX=y.
Oh I was using kvm-coco-queue. Thanks for pointing out.
Quoted 32 lines
Quoted 20 lines
And one more thing:
With the above patch, we still have below code in
vt_init():/* * TDX and VMX have different vCPU structures. Calculate the * maximum size/align so that kvm_init() can use the larger * values to create the kmem_vcpu_cache. */ vcpu_size = sizeof(struct vcpu_vmx); vcpu_align = __alignof__(struct vcpu_vmx); if (enable_tdx) { vcpu_size = max_t(unsigned, vcpu_size, sizeof(struct vcpu_tdx)); vcpu_align = max_t(unsigned, vcpu_align, __alignof__(struct vcpu_tdx)); kvm_caps.supported_vm_types |= BIT(KVM_X86_TDX_VM); }Quoted 7 lines
It's kinda ugly too IMHO.
Since we already have @vm_size in
kvm_x86_ops, how about also addingvcpu_sizeandvcpu_alignto it? Then they can be treated in the same way asvm_sizefor TDX.They are not needed for SVM, but it doesn't hurt that much?
I'd rather not.
vt_init()already needs to be aware of TDX, e.g. to call intotdx_bringup()in the first place. Shoving state intokvm_x86_opsthat is only ever used invt_init()(an__initfunction at that) isn't a net positive.
Quoted 8 lines
Putting the fields in
kvm_x86_init_opswould be better, but I still don't think the complexity and indirection is justified. Bleeding gory TDX details into the common code is undesirable, but I don't see the size ofvcpu_tdxor the fact that enable_tdx==true meansKVM_X86_TDX_VMis supported as being information with hiding.kvm_x86_ops.vm_size is a means to an end. E.g. if
kvm_vcpu_cachedidn't exist, then we'd probably want/need kvm_x86_ops.vcpu_size, but it does exist, so...
Sure. Thanks for clarification.
Sean Christopherson
maintainerRe: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 47 lines · Kai Huang
Quoted 43 lines · Sean Christopherson
Quoted 33 lines · Kai Huang
Quoted 32 lines · Sean Christopherson
(quoted diff) +14
-4+static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) { struct page *page = pfn_to_page(pfn); int ret;@@ -3507,10 +3507,14 @@ int __init tdx_bringup(void) r = __tdx_bringup(); if (r) { /* - * Disable TDX only but don't fail to load module if - * the TDX module could not be loaded. No need to print - * message saying "module is not loaded" because it was - * printed when the first SEAMCALL failed. + * Disable TDX only but don't fail to load module if the TDX + * module could not be loaded. No need to print message saying + * "module is not loaded" because it was printed when the first + * SEAMCALL failed. Don't bother unwinding the S-EPT hooks or + * vm_size, as kvm_x86_ops have already been finalized (and are + * intentionally not exported). The S-EPT code is unreachable, + * and allocating a few more bytes per VM in a should-be-rare + * failure scenario is a non-issue. */ if (r == -ENODEV) goto success_disable_tdx;@@ -3524,3 +3528,19 @@ int __init tdx_bringup(void) enable_tdx = 0; return 0; } + + +void __init tdx_hardware_setup(void) +{Quoted 8 lines
Review int __init tdx_bringup(void) · line 3538
Quoted 4 lines
(quoted diff) +4
-0+ /* + * Note, if the TDX module can't be loaded, KVM TDX support will be + * disabled but KVM will continue loading (see tdx_bringup()). + */This comment seems a little bit weird to me. I think what you meant here is the @vm_size and those S-EPT ops are not unwound while TDX cannot be brought up but KVM is still loaded.
Quoted 2 lines
This comment is weird? Or the one in
tdx_bringup()is weird?
Quoted 7 lines
I definitely agree
tdx_bringup()is weird :-)Quoted 3 lines
The sole intent of _this_ comment is to clarify that KVM could still end up running load with TDX disabled.
But this behaviour itself doesn't mean anything,
I disagree. The overwhelming majority of code in KVM expects that either the associated feature will be fully enabled, or KVM will abort the overall flow, e.g. refuse to load, fail vCPU/VM creation, etc.
Continuing on is very exceptional IMO, and warrants a comment.
Quoted 1 line
e.g., if we export
kvm_x86_ops, we could unwind them.
Maaaybe. I mean, yes, we could fully unwind kvm_x86_ops, but doing so would make the overall code far more brittle. E.g. simply updating kvm_x86_ops won't suffice, as the static_calls also need to be patched, and we would have to be very careful not to touch anything in kvm_x86_ops that might have been consumed between here and the call to tdx_bringup().
Quoted 4 lines
So without mentioning "those are not unwound", it doesn't seem useful to me.
But it does have "(see
tdx_bringup())" at the end, so OK to me. I guess I just wish it could be more verbose.
Yeah, redirecting to another comment isn't a great experience for readers, but I don't want to duplicate the explanation and details because that risks creating stale and/or contradicting comments in the future, and in general increases the maintenance cost (small though it should be in this case).
Re: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 54 lines · Sean Christopherson
Quoted 53 lines · Kai Huang
Quoted 43 lines · Sean Christopherson
Quoted 33 lines · Kai Huang
Quoted 32 lines · Sean Christopherson
(quoted diff) +14
-4+static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, + enum pg_level level, kvm_pfn_t pfn) { struct page *page = pfn_to_page(pfn); int ret;@@ -3507,10 +3507,14 @@ int __init tdx_bringup(void) r = __tdx_bringup(); if (r) { /* - * Disable TDX only but don't fail to load module if - * the TDX module could not be loaded. No need to print - * message saying "module is not loaded" because it was - * printed when the first SEAMCALL failed. + * Disable TDX only but don't fail to load module if the TDX + * module could not be loaded. No need to print message saying + * "module is not loaded" because it was printed when the first + * SEAMCALL failed. Don't bother unwinding the S-EPT hooks or + * vm_size, as kvm_x86_ops have already been finalized (and are + * intentionally not exported). The S-EPT code is unreachable, + * and allocating a few more bytes per VM in a should-be-rare + * failure scenario is a non-issue. */ if (r == -ENODEV) goto success_disable_tdx;@@ -3524,3 +3528,19 @@ int __init tdx_bringup(void) enable_tdx = 0; return 0; } + + +void __init tdx_hardware_setup(void) +{Quoted 8 lines
Review int __init tdx_bringup(void) · line 3538
Quoted 4 lines
(quoted diff) +4
-0+ /* + * Note, if the TDX module can't be loaded, KVM TDX support will be + * disabled but KVM will continue loading (see tdx_bringup()). + */This comment seems a little bit weird to me. I think what you meant here is the @vm_size and those S-EPT ops are not unwound while TDX cannot be brought up but KVM is still loaded.
Quoted 2 lines
This comment is weird? Or the one in
tdx_bringup()is weird?I definitely agree
tdx_bringup()is weird :-)Quoted 3 lines
The sole intent of _this_ comment is to clarify that KVM could still end up running load with TDX disabled.
Quoted 7 lines
Quoted 1 line
But this behaviour itself doesn't mean anything,
I disagree. The overwhelming majority of code in KVM expects that either the associated feature will be fully enabled, or KVM will abort the overall flow, e.g. refuse to load, fail vCPU/VM creation, etc.
Continuing on is very exceptional IMO, and warrants a comment.
I see.
Quoted 8 lines
Quoted 2 lines
e.g., if we export
kvm_x86_ops, we could unwind them.Maaaybe. I mean, yes, we could fully unwind
kvm_x86_ops, but doing so would make the overall code far more brittle. E.g. simply updatingkvm_x86_opswon't suffice, as thestatic_callsalso need to be patched, and we would have to be very careful not to touch anything inkvm_x86_opsthat might have been consumed between here and the call totdx_bringup().
Right. Maybe exporting kvm_ops_update() is better.
Quoted 10 lines
Quoted 5 lines
So without mentioning "those are not unwound", it doesn't seem useful to me.
But it does have "(see
tdx_bringup())" at the end, so OK to me. I guess I just wish it could be more verbose.Yeah, redirecting to another comment isn't a great experience for readers, but I don't want to duplicate the explanation and details because that risks creating stale and/or contradicting comments in the future, and in general increases the maintenance cost (small though it should be in this case).
Sure :-)
Sean Christopherson
maintainerRe: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 10 lines · Kai Huang
Quoted 7 lines · Sean Christopherson
Quoted 1 line
e.g., if we export
kvm_x86_ops, we could unwind them.Maaaybe. I mean, yes, we could fully unwind
kvm_x86_ops, but doing so would make the overall code far more brittle. E.g. simply updatingkvm_x86_opswon't suffice, as thestatic_callsalso need to be patched, and we would have to be very careful not to touch anything inkvm_x86_opsthat might have been consumed between here and the call totdx_bringup().Right. Maybe exporting
kvm_ops_update()is better.
A bit, but KVM would still need to be careful not to modify the parts of vt_x86_ops that have already been consumed.
While I agree that leaving TDX breadcrumbs in kvm_x86_ops when TDX is disabled is undesirable, the behavior is known, i.e. we know exactly what TDX state is being left behind. And failure to load the TDX Module should be very, very rare for any setup that is actually trying to enable TDX.
Whereas providing a way to modify kvm_x86_ops creates the possibility for "bad" updates. KVM's initialization code is a lot like the kernel's boot code (and probably most bootstrapping code): it's inherently fragile because avoiding dependencies is practically impossible.
E.g. I ran into a relevant ordering problem[*] just a few days ago, where checking for VMX capabilities during PMU initialization always failed because the VMCS config hadn't yet been parsed. Those types of bugs are especially dangerous because they're very easy to overlook when modifying existing code, e.g. the only sign that anything is broken is an optional feature being missing.
Re: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Kai Huang
Quoted 7 lines · Sean Christopherson
Quoted 1 line
e.g., if we export
kvm_x86_ops, we could unwind them.Maaaybe. I mean, yes, we could fully unwind
kvm_x86_ops, but doing so would make the overall code far more brittle. E.g. simply updatingkvm_x86_opswon't suffice, as thestatic_callsalso need to be patched, and we would have to be very careful not to touch anything inkvm_x86_opsthat might have been consumed between here and the call totdx_bringup().Right. Maybe exporting
kvm_ops_update()is better.
Quoted 7 lines
A bit, but KVM would still need to be careful not to modify the parts of
vt_x86_opsthat have already been consumed.While I agree that leaving TDX breadcrumbs in
kvm_x86_opswhen TDX is disabled is undesirable, the behavior is known, i.e. we know exactly what TDX state is being left behind. And failure to load the TDX Module should be very, very rare for any setup that is actually trying to enable TDX.
This is true. Agreed.
Quoted 6 lines
Whereas providing a way to modify
kvm_x86_opscreates the possibility for "bad" updates. KVM's initialization code is a lot like the kernel's boot code (and probably most bootstrapping code): it's inherently fragile because avoiding dependencies is practically impossible.
Quoted 7 lines
E.g. I ran into a relevant ordering problem[*] just a few days ago, where checking for VMX capabilities during PMU initialization always failed because the VMCS config hadn't yet been parsed. Those types of bugs are especially dangerous because they're very easy to overlook when modifying existing code, e.g. the only sign that anything is broken is an optional feature being missing.
Right. No argument around this. I agree if there are multiple features wanting to update then it could be dangerous. Thanks for the insight :-)
Sean Christopherson
maintainerRe: [PATCH v3 2/3] KVM: x86: Use kvzalloc() to allocate VM struct
In reply to Sean Christophersonarch/x86/kvm/x86.h struct kvm_host_values { · line 76
Quoted 12 lines · Sean Christopherson
arch/x86/kvm/x86.h +12
-0+/* +* Assert that "struct kvm_{svm,vmx,tdx}" is an order-0 or order-1 allocation. +* Spilling over to an order-2 allocation isn't fundamentally problematic, but +* isn't expected to happen in the foreseeable future (O(years)). Assert that +* the size is an order-0 allocation when ignoring the memslot hash tables, to +* help detect and debug unexpected size increases. +*/ +#define KVM_SANITY_CHECK_VM_STRUCT_SIZE (x) \ +do { \ + BUILD_BUG_ON(get_order(sizeof(struct x) - SIZE_OF_MEMSLOTS_HASHTABLE) && \ + !IS_ENABLED(CONFIG_DEBUG_KERNEL) && !IS_ENABLED(CONFIG_KASAN)); \ + BUILD_BUG_ON(get_order(sizeof(struct x)) < 2 && \
Ugh, I jinxed myself. My severe ineptitude and inability to test persists. I inverted this check, it should be:
get_order(sizeof(struct x)) >= 2
Quoted 9 lines
arch/x86/kvm/x86.h +3
-0+ !IS_ENABLED(CONFIG_DEBUG_KERNEL) && !IS_ENABLED(CONFIG_KASAN)); \ +} while (0) +#define KVM_NESTED_VMENTER_CONSISTENCY_CHECK (consistency_check) \ ({ \ bool failed = (consistency_check); \Signature · 1 line
2.49.0.1112.g889b7c5bd8-goog
Sean Christopherson
maintainerPATCH v3 3/3 KVM: x86/mmu: Defer allocation of shadow MMU's hashed page list
In reply to Sean ChristophersonWhen the TDP MMU is enabled, i.e. when the shadow MMU isn't used until a nested TDP VM is run, defer allocation of the array of hashed lists used to track shadow MMU pages until the first shadow root is allocated.
Setting the list outside of mmu_lock is safe, as concurrent readers must hold mmu_lock in some capacity, shadow pages can only be added (or removed) from the list when mmu_lock is held for write, and tasks that are creating a shadow root are serialized by slots_arch_lock. I.e. it's impossible for the list to become non-empty until all readers go away, and so readers are guaranteed to see an empty list even if they make multiple calls to kvm_get_mmu_page_hash() in a single mmu_lock critical section.
Use {WRITE/READ}_ONCE to set/get the list when mmu_lock isn't held for write, out of an abundance of paranoia; no sane compiler should tear the store or load, but it's technically possible.
--- arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 10 deletions(-)
arch/x86/kvm/mmu/mmu.c +50 -10
@@ -1983,14 +1983,33 @@ static bool sp_has_gptes(struct kvm_mmu_page *sp)return true; } +static __ro_after_init HLIST_HEAD (empty_page_hash); + +static struct hlist_head *kvm_get_mmu_page_hash (struct kvm *kvm,gfn_t gfn) +{ +/* +* Load mmu_page_hash from memory exactly once, as it's set at runtime +* outside of mmu_lock when the TDP MMU is enabled, i.e. when the hash +* table of shadow pages isn't needed unless KVM needs to shadow L1's +* TDP for an L2 guest. +*/ +struct hlist_head *page_hash =READ_ONCE (kvm->arch.mmu_page_hash); + +lockdep_assert_held (&kvm->mmu_lock); + +if (!page_hash) +return &empty_page_hash; + +return &page_hash[kvm_page_table_hashfn (gfn)]; +} +#define for_each_valid_sp (_kvm, _sp, _list) \ hlist_for_each_entry(_sp, _list, hash_link) \ if (is_obsolete_sp((_kvm), (_sp))) { \ } else#define for_each_gfn_valid_sp_with_gptes (_kvm, _sp, _gfn) \ - for_each_valid_sp(_kvm, _sp, \ - &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)]) \ + for_each_valid_sp(_kvm, _sp, kvm_get_mmu_page_hash(_kvm, _gfn)) \ if ((_sp)->gfn != (_gfn) || !sp_has_gptes(_sp)) {} elsestatic bool kvm_sync_page_check (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp)
@@ -2358,6 +2377,12 @@ static struct kvm_mmu_page *__kvm_mmu_get_shadow_page(struct kvm *kvm,struct kvm_mmu_page *sp;bool created = false; +/* +* No need for READ_ONCE(), unlike in kvm_get_mmu_page_hash(), because +* mmu_page_hash must be set prior to creating the first shadow root, +* i.e. reaching this point is fully serialized by slots_arch_lock. +*/ +BUG_ON (!kvm->arch.mmu_page_hash); sp_list = &kvm->arch.mmu_page_hash[kvm_page_table_hashfn (gfn)]; sp =kvm_mmu_find_shadow_page (kvm, vcpu, gfn, sp_list, role);
@@ -3886,11 +3911,21 @@ static int kvm_mmu_alloc_page_hash(struct kvm *kvm) { typeof(kvm->arch.mmu_page_hash) h; +if (kvm->arch.mmu_page_hash) +return 0 ; + h =kvcalloc (KVM_NUM_MMU_PAGES ,sizeof (*h),GFP_KERNEL_ACCOUNT );if (!h)return -ENOMEM ; - kvm->arch.mmu_page_hash = h; +/* +* Write mmu_page_hash exactly once as there may be concurrent readers, +* e.g. to check for shadowed PTEs in mmu_try_to_unsync_pages(). Note, +* mmu_lock must be held for write to add (or remove) shadow pages, and +* so readers are guaranteed to see an empty list for their current +* mmu_lock critical section. +*/ +WRITE_ONCE (kvm->arch.mmu_page_hash, h);return 0 ; }
@@ -3913,9 +3948,13 @@ static int mmu_first_shadow_root_alloc(struct kvm *kvm)if (kvm_shadow_root_allocated (kvm)) goto out_unlock; + r =kvm_mmu_alloc_page_hash (kvm); +if (r) + goto out_unlock; +/* -* Check if anything actually needs to be allocated, e.g. all metadata -* will be allocated upfront if TDP is disabled. +* Check if memslot metadata actually needs to be allocated, e.g. all +* metadata will be allocated upfront if TDP is disabled. */ if (kvm_memslots_have_rmaps (kvm) &&kvm_page_track_write_tracking_enabled (kvm))
@@ -6696,12 +6735,13 @@ int kvm_mmu_init_vm(struct kvm *kvm)INIT_LIST_HEAD (&kvm->arch.possible_nx_huge_pages);spin_lock_init (&kvm->arch.mmu_unsync_pages_lock); - r =kvm_mmu_alloc_page_hash (kvm); -if (r) -return r; - -if (tdp_mmu_enabled) +if (tdp_mmu_enabled) {kvm_mmu_init_tdp_mmu (kvm); + }else { + r =kvm_mmu_alloc_page_hash (kvm); +if (r) +return r; + } kvm->arch.split_page_header_cache.kmem_cache = mmu_page_header_cache; kvm->arch.split_page_header_cache.gfp_zero = __GFP_ZERO;
Signature · 1 line
2.49.0.1112.g889b7c5bd8-goog
Paolo Bonzini
maintainerRe: [PATCH v3 3/3] KVM: x86/mmu: Defer allocation of shadow MMU's hashed page list
Quoted 28 lines · Sean Christopherson
When the TDP MMU is enabled, i.e. when the shadow MMU isn't used until a nested TDP VM is run, defer allocation of the array of hashed lists used to track shadow MMU pages until the first shadow root is allocated.
Setting the list outside of
mmu_lockis safe, as concurrent readers must holdmmu_lockin some capacity, shadow pages can only be added (or removed) from the list whenmmu_lockis held for write, and tasks that are creating a shadow root are serialized byslots_arch_lock. I.e. it's impossible for the list to become non-empty until all readers go away, and so readers are guaranteed to see an empty list even if they make multiple calls tokvm_get_mmu_page_hash()in a singlemmu_lockcritical section.Use {WRITE/READ}_ONCE to set/get the list when
mmu_lockisn't held for write, out of an abundance of paranoia; no sane compiler should tear the store or load, but it's technically possible.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++++++++------- 1 file changed, 50 insertions(+), 10 deletions(-)arch/x86/kvm/mmu/mmu.c +0
-0@@ -1983,14 +1983,33 @@ static bool sp_has_gptes(struct kvm_mmu_page *sp)return true; }
arch/x86/kvm/mmu/mmu.c static bool sp_has_gptes(struct kvm_mmu_page *sp) · line 1994
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +9
-0+static __ro_after_init HLIST_HEAD (empty_page_hash); + +static struct hlist_head *kvm_get_mmu_page_hash (struct kvm *kvm,gfn_t gfn) +{ +/* +* Load mmu_page_hash from memory exactly once, as it's set at runtime +* outside of mmu_lock when the TDP MMU is enabled, i.e. when the hash +* table of shadow pages isn't needed unless KVM needs to shadow L1's +* TDP for an L2 guest.
Avoid double negations, for example "i.e. when shadow paging and the mmu_page_hash is only needed for a L1 hypervisor's TDP".
arch/x86/kvm/mmu/mmu.c static bool sp_has_gptes(struct kvm_mmu_page *sp) · line 1996
Quoted 2 lines
arch/x86/kvm/mmu/mmu.c +2
-0+*/ +struct hlist_head *page_hash =READ_ONCE (kvm->arch.mmu_page_hash);
This READ_ONCE is more for the (now implicit) smp_read_barrier_depends() than for "loading exactly once". Might as well use smp_load_acquire() since it's free on x86.
arch/x86/kvm/mmu/mmu.c static bool sp_has_gptes(struct kvm_mmu_page *sp) · line 1998
Quoted 1 line
arch/x86/kvm/mmu/mmu.c +1
-0+lockdep_assert_held (&kvm->mmu_lock);
Move this comment here from kvm_mmu_alloc_page_hash():
/* mmu_lock must be held for write to add (or remove) shadow * pages, and so readers are guaranteed to see an empty list for * their current mmu_lock critical section. */
Quoted 41 lines
arch/x86/kvm/mmu/mmu.c +16
-2+if (!page_hash) +return &empty_page_hash; + +return &page_hash[kvm_page_table_hashfn (gfn)]; +} +#define for_each_valid_sp (_kvm, _sp, _list) \ hlist_for_each_entry(_sp, _list, hash_link) \ if (is_obsolete_sp((_kvm), (_sp))) { \ } else#define for_each_gfn_valid_sp_with_gptes (_kvm, _sp, _gfn) \ - for_each_valid_sp(_kvm, _sp, \ - &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)]) \ + for_each_valid_sp(_kvm, _sp, kvm_get_mmu_page_hash(_kvm, _gfn)) \ if ((_sp)->gfn != (_gfn) || !sp_has_gptes(_sp)) {} elsestatic bool kvm_sync_page_check (struct kvm_vcpu *vcpu,struct kvm_mmu_page *sp)@@ -2358,6 +2377,12 @@ static struct kvm_mmu_page *__kvm_mmu_get_shadow_page(struct kvm *kvm,struct kvm_mmu_page *sp;bool created = false; +/* +* No need for READ_ONCE(), unlike in kvm_get_mmu_page_hash(), because +* mmu_page_hash must be set prior to creating the first shadow root, +* i.e. reaching this point is fully serialized by slots_arch_lock. +*/ +BUG_ON (!kvm->arch.mmu_page_hash); sp_list = &kvm->arch.mmu_page_hash[kvm_page_table_hashfn (gfn)]; sp =kvm_mmu_find_shadow_page (kvm, vcpu, gfn, sp_list, role);@@ -3886,11 +3911,21 @@ static int kvm_mmu_alloc_page_hash(struct kvm *kvm) { typeof(kvm->arch.mmu_page_hash) h; +if (kvm->arch.mmu_page_hash) +return 0 ; + h =kvcalloc (KVM_NUM_MMU_PAGES ,sizeof (*h),GFP_KERNEL_ACCOUNT );if (!h)return -ENOMEM ;
arch/x86/kvm/mmu/mmu.c static int kvm_mmu_alloc_page_hash(struct kvm *kvm) · line 3928
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +8
-1- kvm->arch.mmu_page_hash = h; +/* +* Write mmu_page_hash exactly once as there may be concurrent readers, +* e.g. to check for shadowed PTEs in mmu_try_to_unsync_pages(). Note, +* mmu_lock must be held for write to add (or remove) shadow pages, and +* so readers are guaranteed to see an empty list for their current +* mmu_lock critical section. +*/ +WRITE_ONCE (kvm->arch.mmu_page_hash, h);
Use smp_store_release here (unlike READ_ONCE(), it's technically incorrect to use WRITE_ONCE() here!), with a remark that it pairs with kvm_get_mmu_page_hash(). That's both more accurate and leads to a better comment than "write exactly once".
Paolo
Sean Christopherson
maintainerRe: [PATCH v3 3/3] KVM: x86/mmu: Defer allocation of shadow MMU's hashed page list
Quoted 12 lines · Paolo Bonzini
arch/x86/kvm/mmu/mmu.c static int kvm_mmu_alloc_page_hash(struct kvm *kvm) · line 3928
Quoted 8 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +8
-0+/* +* Write mmu_page_hash exactly once as there may be concurrent readers, +* e.g. to check for shadowed PTEs in mmu_try_to_unsync_pages(). Note, +* mmu_lock must be held for write to add (or remove) shadow pages, and +* so readers are guaranteed to see an empty list for their current +* mmu_lock critical section. +*/ +WRITE_ONCE (kvm->arch.mmu_page_hash, h);Use
smp_store_releasehere (unlikeREAD_ONCE(), it's technically incorrect to useWRITE_ONCE()here!),
Can you elaborate why? Due to my x86-centric life, my memory ordering knowledge is woefully inadequate.
Quoted 2 lines
with a remark that it pairs with
kvm_get_mmu_page_hash(). That's both more accurate and leads to a better comment than "write exactly once".
Re: [PATCH v3 3/3] KVM: x86/mmu: Defer allocation of shadow MMU's hashed page list
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Paolo Bonzini
Quoted 8 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +8
-0+/* +* Write mmu_page_hash exactly once as there may be concurrent readers, +* e.g. to check for shadowed PTEs in mmu_try_to_unsync_pages(). Note, +* mmu_lock must be held for write to add (or remove) shadow pages, and +* so readers are guaranteed to see an empty list for their current +* mmu_lock critical section. +*/ +WRITE_ONCE (kvm->arch.mmu_page_hash, h);
Quoted 5 lines
Quoted 2 lines
Use
smp_store_releasehere (unlikeREAD_ONCE(), it's technically incorrect to useWRITE_ONCE()here!),Can you elaborate why? Due to my x86-centric life, my memory ordering knowledge is woefully inadequate.
The compiler must be prohibited from reordering stores preceding this WRITE_ONCE() to after it.
In reality, the only stores that matter will be from within kvcalloc(), and the important bits of it will not be inlined, so it's unlikely that the compiler would actually do such reordering. But it's nonetheless allowed. :) barrier() is precisely what is needed to prohibit this; smp_store_release() on x86 is merely barrier() + WRITE_ONCE().
Semantically, smp_store_release() is what you mean to write, as Paolo said. We're not really *only* preventing torn accesses, we also need to ensure that any threads that read kvm->arch.mmu_page_hash can actually use that result (i.e., that all the stores from the kvcalloc() are visible). This sounds a little bit weird for x86 code, but compiler reordering is still a possibility.
And I also agree with what Paolo said about smp_load_acquire(). :)
Thanks Paolo. Please correct me if I'm wrong above.
Quoted 3 lines
Quoted 2 lines
with a remark that it pairs with
kvm_get_mmu_page_hash(). That's both more accurate and leads to a better comment than "write exactly once".
Sean Christopherson
maintainerRe: [PATCH v3 3/3] KVM: x86/mmu: Defer allocation of shadow MMU's hashed page list
Quoted 29 lines · James Houghton
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Paolo Bonzini
Quoted 8 lines · Sean Christopherson
arch/x86/kvm/mmu/mmu.c +8
-0+/* +* Write mmu_page_hash exactly once as there may be concurrent readers, +* e.g. to check for shadowed PTEs in mmu_try_to_unsync_pages(). Note, +* mmu_lock must be held for write to add (or remove) shadow pages, and +* so readers are guaranteed to see an empty list for their current +* mmu_lock critical section. +*/ +WRITE_ONCE (kvm->arch.mmu_page_hash, h);Quoted 5 lines
Quoted 2 lines
Use
smp_store_releasehere (unlikeREAD_ONCE(), it's technically incorrect to useWRITE_ONCE()here!),Can you elaborate why? Due to my x86-centric life, my memory ordering knowledge is woefully inadequate.
The compiler must be prohibited from reordering stores preceding this
WRITE_ONCE()to after it.In reality, the only stores that matter will be from within
kvcalloc(), and the important bits of it will not be inlined, so it's unlikely that the compiler would actually do such reordering. But it's nonetheless allowed. :)barrier()is precisely what is needed to prohibit this;smp_store_release()on x86 is merelybarrier()+WRITE_ONCE().
Quoted 5 lines
Semantically,
smp_store_release()is what you mean to write, as Paolo said. We're not really *only* preventing torn accesses, we also need to ensure that any threads that readkvm->arch.mmu_page_hashcan actually use that result (i.e., that all the stores from thekvcalloc()are visible).
Ah, that's what I was missing. It's not KVM's stores that are theoretically problematic, it's the zeroing of kvm->arch.mmu_page_hash that needs protection.
Thanks!
Quoted 10 lines
This sounds a little bit weird for x86 code, but compiler reordering is still a possibility.
And I also agree with what Paolo said about
smp_load_acquire(). :)Thanks Paolo. Please correct me if I'm wrong above.
Quoted 3 lines
Quoted 2 lines
with a remark that it pairs with
kvm_get_mmu_page_hash(). That's both more accurate and leads to a better comment than "write exactly once".