KVM: x86/mmu: TDX post-populate cleanups
Subsystems (4)
Sean Christopherson
maintainercover KVM: x86/mmu: TDX post-populate cleanups
This is a largely untested series to do most of what was discussed in the thread regarding locking issues between gmem and TDX's post-populate hook[*], with more than a few side quests thrown in as I was navigating through the code to try to figure out how best to eliminate the copy_from_user() from sev_gmem_post_populate(), which has the same locking problem (copying from a userspace address can fault and in theory trigger the same problematic path, I think).
Notably absent is the extraction of copy_from_user() from sev_gmem_post_populate() to kvm_gmem_populate(). I've had this on my todo list for a few weeks now, and haven't been able to focus on it for long enough to get something hammered out, and with KVM Forum on the horizon, I don't anticipate getting 'round to it within the next month (if not much longer).
The thing that stymied me is what to do if snp_launch_update() is passed in a huge batch of pages. I waffled between doing a slow one-at-a-time approach and a batched approached, and got especially stuck when trying to remember and/or figure out how that handling would interact with hugepage support in SNP in particular.
If anyone wants to tackle that project, the one thing change I definitely think we should do is change the post_populate() callback to operate on exactly one page. KVM_SEV_SNP_LAUNCH_UPDATE allows for partial progress, i.e. KVM's ABI doesn't require it to unwind a batch if adding a page fails. If we take advantage of that, then sev_gmem_post_populate() will be a bit simpler (though I wouldn't go so far as to call it "simple").
RFC as this is compile tested only (mostly due to lack of access to a TDX capable system, but also due to lack of cycles).
[*] http://lore.kernel.org/all/[email protected]
Sean Christopherson (12):
KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings
KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU Revert "KVM: x86/tdp_mmu: Add a helper function to walk down the TDP
MMU"
KVM: x86/mmu: Rename kvm_tdp_map_page() to kvm_tdp_prefault_page()
KVM: TDX: Drop superfluous page pinning in S-EPT management
KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte()
KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
KVM: TDX: Assert that slots_lock is held when nr_premapped is accessed
KVM: TDX: Track nr_premapped as an "unsigned long", not an
"atomic64_t"
KVM: TDX: Rename nr_premapped to nr_pending_tdh_mem_page_adds
arch/x86/kvm/mmu.h | 3 +- arch/x86/kvm/mmu/mmu.c | 66 ++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 37 ++--------- arch/x86/kvm/vmx/tdx.c | 123 +++++++++++++------------------------ arch/x86/kvm/vmx/tdx.h | 9 ++- 5 files changed, 117 insertions(+), 121 deletions(-)
base-commit: 196d9e72c4b0bd68b74a4ec7f52d248f37d0f030
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Sean Christopherson
maintainerRFC PATCH 1/12 KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings
Drop TDX's sanity check that an S-EPT mapping isn't zapped between creating said mapping and doing TDH.MEM.PAGE.ADD, as the check is simultaneously superfluous and incomplete. Per commit 2608f1057601 ("KVM: x86/tdp_mmu:
Add a helper function to walk down the TDP MMU"), the justification for introducing kvm_tdp_mmu_gpa_is_mapped() was to check that the target gfn was pre-populated, with a link that points to this snippet:
: > One small question:
: >
: > What if the memory region passed to KVM_TDX_INIT_MEM_REGION hasn't been pre- : > populated? If we want to make KVM_TDX_INIT_MEM_REGION work with these regions, : > then we still need to do the real map. Or we can make KVM_TDX_INIT_MEM_REGION : > return error when it finds the region hasn't been pre-populated? :
: Return an error. I don't love the idea of bleeding so many TDX details into : userspace, but I'm pretty sure that ship sailed a long, long time ago.
But that justification makes little sense for the final code, as simply doing TDH.MEM.PAGE.ADD without a paranoid sanity check will return an error if the S-EPT mapping is invalid (as evidenced by the code being guarded with CONFIG_KVM_PROVE_MMU=y).
The sanity check is also incomplete in the sense that mmu_lock is dropped between the check and TDH.MEM.PAGE.ADD, i.e. will only detect KVM bugs that zap SPTEs in a very specific window.
Removing the sanity check will allow removing kvm_tdp_mmu_gpa_is_mapped(), which has no business being exposed to vendor code.
--- arch/x86/kvm/vmx/tdx.c | 14 -------------- 1 file changed, 14 deletions(-)
arch/x86/kvm/vmx/tdx.c +0 -14
@@ -3175,20 +3175,6 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,if (ret <0 ) goto out; -/* -* The private mem cannot be zapped after kvm_tdp_map_page() -* because all paths are covered by slots_lock and the -* filemap invalidate lock. Check that they are indeed enough. -*/ -if (IS_ENABLED (CONFIG_KVM_PROVE_MMU )) { - scoped_guard(read_lock, &kvm->mmu_lock) { -if (KVM_BUG_ON (!kvm_tdp_mmu_gpa_is_mapped (vcpu, gpa), kvm)) { - ret = -EIO ; - goto out; - } - } - } - ret =0 ; err =tdh_mem_page_add (&kvm_tdx->td, gpa,pfn_to_page (pfn), src_page, &entry, &level_state);
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 01/12] KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings
Quoted 17 lines · Sean Christopherson
Drop TDX's sanity check that an S-EPT mapping isn't zapped between creating said mapping and doing TDH.MEM.PAGE.ADD, as the check is simultaneously superfluous and incomplete. Per commit
2608f1057601("KVM: x86/tdp_mmu:
Add a helper function to walk down the TDP MMU"), the justification for introducingkvm_tdp_mmu_gpa_is_mapped()was to check that the target gfn was pre-populated, with a link that points to this snippet:: > One small question:
: >
: > What if the memory region passed toKVM_TDX_INIT_MEM_REGIONhasn't been pre- : > populated? If we want to makeKVM_TDX_INIT_MEM_REGIONwork with these regions, : > then we still need to do the real map. Or we can makeKVM_TDX_INIT_MEM_REGION: > return error when it finds the region hasn't been pre-populated? :
: Return an error. I don't love the idea of bleeding so many TDX details into : userspace, but I'm pretty sure that ship sailed a long, long time ago.
Quoted 4 lines
But that justification makes little sense for the final code, as simply doing TDH.MEM.PAGE.ADD without a paranoid sanity check will return an error if the S-EPT mapping is invalid (as evidenced by the code being guarded with
CONFIG_KVM_PROVE_MMU=y).
Checking of kvm_tdp_mmu_gpa_is_mapped() was intended to detect unexpected zaps like kvm_zap_gfn_range() between kvm_tdp_map_page() and tdh_mem_page_add()? In that case, TDH.MEM.PAGE.ADD would succeed without any error.
But as you said, the read mmu_lock is dropped before tdh_mem_page_add(). Moreover, it still cannot guard against atomic zaps.
As zaps between kvm_tdp_map_page() and tdh_mem_page_add() could still be detectable through the incorrect value of nr_premapped in the end, dropping the checks of kvm_tdp_mmu_gpa_is_mapped() looks good.
Quoted 40 lines
The sanity check is also incomplete in the sense that
mmu_lockis dropped between the check and TDH.MEM.PAGE.ADD, i.e. will only detect KVM bugs that zap SPTEs in a very specific window.Removing the sanity check will allow removing
kvm_tdp_mmu_gpa_is_mapped(), which has no business being exposed to vendor code.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 14 -------------- 1 file changed, 14 deletions(-)arch/x86/kvm/vmx/tdx.c +0
-14@@ -3175,20 +3175,6 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,if (ret <0 ) goto out; -/* -* The private mem cannot be zapped after kvm_tdp_map_page() -* because all paths are covered by slots_lock and the -* filemap invalidate lock. Check that they are indeed enough. -*/ -if (IS_ENABLED (CONFIG_KVM_PROVE_MMU )) { - scoped_guard(read_lock, &kvm->mmu_lock) { -if (KVM_BUG_ON (!kvm_tdp_mmu_gpa_is_mapped (vcpu, gpa), kvm)) { - ret = -EIO ; - goto out; - } - } - } - ret =0 ; err =tdh_mem_page_add (&kvm_tdx->td, gpa,pfn_to_page (pfn), src_page, &entry, &level_state);Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 01/12] KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings
In reply to Sean ChristophersonQuoted 22 lines · Sean Christopherson
Drop TDX's sanity check that an S-EPT mapping isn't zapped between creating said mapping and doing TDH.MEM.PAGE.ADD, as the check is simultaneously superfluous and incomplete. Per commit
2608f1057601("KVM: x86/tdp_mmu:
Add a helper function to walk down the TDP MMU"), the justification for introducingkvm_tdp_mmu_gpa_is_mapped()was to check that the target gfn was pre-populated, with a link that points to this snippet:: > One small question:
: >
: > What if the memory region passed toKVM_TDX_INIT_MEM_REGIONhasn't been pre- : > populated? If we want to makeKVM_TDX_INIT_MEM_REGIONwork with these regions, : > then we still need to do the real map. Or we can makeKVM_TDX_INIT_MEM_REGION: > return error when it finds the region hasn't been pre-populated? :
: Return an error. I don't love the idea of bleeding so many TDX details into : userspace, but I'm pretty sure that ship sailed a long, long time ago.But that justification makes little sense for the final code, as simply doing TDH.MEM.PAGE.ADD without a paranoid sanity check will return an error if the S-EPT mapping is invalid (as evidenced by the code being guarded with
CONFIG_KVM_PROVE_MMU=y).
Quoted 8 lines
The sanity check is also incomplete in the sense that
mmu_lockis dropped between the check and TDH.MEM.PAGE.ADD, i.e. will only detect KVM bugs that zap SPTEs in a very specific window.Removing the sanity check will allow removing
kvm_tdp_mmu_gpa_is_mapped(), which has no business being exposed to vendor code.Signed-off-by Sean Christopherson <[email protected]>
[snip]
Re: [RFC PATCH 01/12] KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings
In reply to Sean ChristophersonQuoted 22 lines · Sean Christopherson
Drop TDX's sanity check that an S-EPT mapping isn't zapped between creating said mapping and doing TDH.MEM.PAGE.ADD, as the check is simultaneously superfluous and incomplete. Per commit
2608f1057601("KVM: x86/tdp_mmu:
Add a helper function to walk down the TDP MMU"), the justification for introducingkvm_tdp_mmu_gpa_is_mapped()was to check that the target gfn was pre-populated, with a link that points to this snippet:: > One small question:
: >
: > What if the memory region passed toKVM_TDX_INIT_MEM_REGIONhasn't been pre- : > populated? If we want to makeKVM_TDX_INIT_MEM_REGIONwork with these regions, : > then we still need to do the real map. Or we can makeKVM_TDX_INIT_MEM_REGION: > return error when it finds the region hasn't been pre-populated? :
: Return an error. I don't love the idea of bleeding so many TDX details into : userspace, but I'm pretty sure that ship sailed a long, long time ago.But that justification makes little sense for the final code, as simply doing TDH.MEM.PAGE.ADD without a paranoid sanity check will return an error if the S-EPT mapping is invalid (as evidenced by the code being guarded with
CONFIG_KVM_PROVE_MMU=y).
Quoted 8 lines
The sanity check is also incomplete in the sense that
mmu_lockis dropped between the check and TDH.MEM.PAGE.ADD, i.e. will only detect KVM bugs that zap SPTEs in a very specific window.Removing the sanity check will allow removing
kvm_tdp_mmu_gpa_is_mapped(), which has no business being exposed to vendor code.Signed-off-by Sean Christopherson <[email protected]>
I guess I asked that small question :-)
Sean Christopherson
maintainerRFC PATCH 2/12 KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
In reply to Sean ChristophersonAdd and use a new API for mapping a private pfn from guest_memfd into the TDP MMU from TDX's post-populate hook instead of partially open-coding the functionality into the TDX code. Sharing code with the pre-fault path sounded good on paper, but it's fatally flawed as simulating a fault loses the pfn, and calling back into gmem to re-retrieve the pfn creates locking problems, e.g. kvm_gmem_populate() already holds the gmem invalidation lock.
Providing a dedicated API will also removing several MMU exports that ideally would not be exposed outside of the MMU, let alone to vendor code. On that topic, opportunistically drop the kvm_mmu_load() export. Leave kvm_tdp_mmu_gpa_is_mapped() alone for now; the entire commit that added kvm_tdp_mmu_gpa_is_mapped() will be removed in the near future.
--- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.c | 10 +++---- 3 files changed, 63 insertions(+), 8 deletions(-)
arch/x86/kvm/mmu.h +1 -0
@@ -259,6 +259,7 @@ extern bool tdp_mmu_enabled;bool kvm_tdp_mmu_gpa_is_mapped (struct kvm_vcpu *vcpu,u64 gpa);int kvm_tdp_map_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code,u8 *level); +int kvm_tdp_mmu_map_private_pfn (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn);static inline bool kvm_memslots_have_rmaps (struct kvm *kvm) {
arch/x86/kvm/mmu/mmu.c +59 -1
72 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.c +3 -7
@@ -3151,15 +3151,12 @@ struct tdx_gmem_post_populate_arg {static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,void __user *src,int order,void *_arg) { -u64 error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS ; -struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);struct tdx_gmem_post_populate_arg *arg = _arg; -struct kvm_vcpu *vcpu = arg->vcpu; +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err, entry, level_state;gpa_t gpa =gfn_to_gpa (gfn); -u8 level =PG_LEVEL_4K ;struct page *src_page;int ret, i; -u64 err, entry, level_state;/* * Get the source page if it has been faulted in. Return failure if the
@@ -3171,7 +3168,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,if (ret !=1 )return -ENOMEM ; - ret =kvm_tdp_map_page (vcpu, gpa, error_code, &level); + ret =kvm_tdp_mmu_map_private_pfn (arg->vcpu, gfn, pfn);if (ret <0 ) goto out;
@@ -3234,7 +3231,6 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c !vt_is_tdx_private_gpa(kvm, region.gpa + (region.nr_pages << PAGE_SHIFT) - 1))return -EINVAL ; -kvm_mmu_reload (vcpu); ret =0 ;while (region.nr_pages) {if (signal_pending (current)) {
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 44 lines · Sean Christopherson
Add and use a new API for mapping a private pfn from
guest_memfdinto the TDP MMU from TDX's post-populate hook instead of partially open-coding the functionality into the TDX code. Sharing code with the pre-fault path sounded good on paper, but it's fatally flawed as simulating a fault loses the pfn, and calling back into gmem to re-retrieve the pfn creates locking problems, e.g.kvm_gmem_populate()already holds the gmem invalidation lock.Providing a dedicated API will also removing several MMU exports that ideally would not be exposed outside of the MMU, let alone to vendor code. On that topic, opportunistically drop the
kvm_mmu_load()export. Leavekvm_tdp_mmu_gpa_is_mapped()alone for now; the entire commit that addedkvm_tdp_mmu_gpa_is_mapped()will be removed in the near future.Cc Michael Roth <[email protected]>Cc Yan Zhao <[email protected]>Cc Ira Weiny <[email protected]>Cc Vishal Annapurve <[email protected]>Cc Rick Edgecombe <[email protected]>Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 60 +++++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/vmx/tdx.c | 10 +++---- 3 files changed, 63 insertions(+), 8 deletions(-)arch/x86/kvm/mmu.h +1
-0@@ -259,6 +259,7 @@ extern bool tdp_mmu_enabled;bool kvm_tdp_mmu_gpa_is_mapped (struct kvm_vcpu *vcpu,u64 gpa);int kvm_tdp_map_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code,u8 *level); +int kvm_tdp_mmu_map_private_pfn (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn);static inline bool kvm_memslots_have_rmaps (struct kvm *kvm) {arch/x86/kvm/mmu/mmu.c +0
-0
arch/x86/kvm/mmu/mmu.c long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · line 4993
Quoted 4 lines
arch/x86/kvm/mmu/mmu.c +1
-0@@ -4990,6 +4990,65 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,return min (range->size, end - range->gpa); } +int kvm_tdp_mmu_map_private_pfn (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn)
As the function starts with kvm_tdp_mmu, can we move it to tdp_mmu.c?
arch/x86/kvm/mmu/mmu.c long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · line 5002
Quoted 9 lines
arch/x86/kvm/mmu/mmu.c +9
-0+{ +struct kvm_page_fault fault = { + .addr =gfn_to_gpa (gfn), + .error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS , + .prefetch = true, + .is_tdp = true, + .nx_huge_page_workaround_enabled =is_nx_huge_page_enabled (vcpu->kvm), + + .max_level =KVM_MAX_HUGEPAGE_LEVEL ,
Looks the kvm_tdp_mmu_map_private_pfn() is only for initial memory mapping, given that ".prefetch = true" and RET_PF_SPURIOUS is not a valid return value.
Then, what about setting
.max_level = PG_LEVEL_4K,
directly?
Otherwise, the "(KVM_BUG_ON(level != PG_LEVEL_4K, kvm)" would be triggered in tdx_sept_set_private_spte().
Quoted 102 lines
arch/x86/kvm/mmu/mmu.c +49
-1+ .req_level = PG_LEVEL_4K, + .goal_level = PG_LEVEL_4K, + .is_private = true, + + .gfn = gfn, + .slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn), + .pfn = pfn, + .map_writable = true, + }; +struct kvm *kvm = vcpu->kvm; +int r; + +lockdep_assert_held (&kvm->slots_lock); + +if (KVM_BUG_ON (!tdp_mmu_enabled, kvm)) +return -EIO ; + +if (kvm_gfn_is_write_tracked (kvm, fault.slot, fault.gfn)) +return -EPERM ; + + r =kvm_mmu_reload (vcpu); +if (r) +return r; + + r =mmu_topup_memory_caches (vcpu, false); +if (r) +return r; + +do { +if (signal_pending (current)) +return -EINTR ; + +if (kvm_test_request (KVM_REQ_VM_DEAD , vcpu)) +return -EIO ; + +cond_resched (); + +guard (read_lock)(&kvm->mmu_lock); + + r =kvm_tdp_mmu_map (vcpu, &fault); + }while (r ==RET_PF_RETRY ); + +if (r !=RET_PF_FIXED ) +return -EIO ; + +return 0 ; +} +EXPORT_SYMBOL_GPL (kvm_tdp_mmu_map_private_pfn); +static void nonpaging_init_context (struct kvm_mmu *context) { context->page_fault = nonpaging_page_fault;@@ -5973,7 +6032,6 @@ int kvm_mmu_load(struct kvm_vcpu *vcpu) out:return r; } -EXPORT_SYMBOL_GPL (kvm_mmu_load);void kvm_mmu_unload (struct kvm_vcpu *vcpu) {arch/x86/kvm/vmx/tdx.c +3
-7@@ -3151,15 +3151,12 @@ struct tdx_gmem_post_populate_arg {static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn,void __user *src,int order,void *_arg) { -u64 error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS ; -struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);struct tdx_gmem_post_populate_arg *arg = _arg; -struct kvm_vcpu *vcpu = arg->vcpu; +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err, entry, level_state;gpa_t gpa =gfn_to_gpa (gfn); -u8 level =PG_LEVEL_4K ;struct page *src_page;int ret, i; -u64 err, entry, level_state;/* * Get the source page if it has been faulted in. Return failure if the @@ -3171,7 +3168,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,if (ret !=1 )return -ENOMEM ; - ret =kvm_tdp_map_page (vcpu, gpa, error_code, &level); + ret =kvm_tdp_mmu_map_private_pfn (arg->vcpu, gfn, pfn);if (ret <0 ) goto out;@@ -3234,7 +3231,6 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c !vt_is_tdx_private_gpa(kvm, region.gpa + (region.nr_pages << PAGE_SHIFT) - 1))return -EINVAL ; -kvm_mmu_reload (vcpu); ret =0 ;while (region.nr_pages) {if (signal_pending (current)) {Signature · 1 line
2.51.0.268.g9569e192d0-goog
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 5 lines · Yan Zhao
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +4
-0+{ + struct kvm_page_fault fault = { + .addr = gfn_to_gpa(gfn), + .error_code = PFERR_GUEST_FINAL_MASK |
PFERR_PRIVATE_ACCESS,
Quoted 8 lines
arch/x86/kvm/mmu/mmu.c long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · line 5002
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ .prefetch = true, + .is_tdp = true, + .nx_huge_page_workaround_enabled =
is_nx_huge_page_enabled(vcpu->kvm),arch/x86/kvm/mmu/mmu.c +2
-0+ + .max_level = KVM_MAX_HUGEPAGE_LEVEL,Looks the
kvm_tdp_mmu_map_private_pfn()is only for initial memory mapping, given that ".prefetch = true" andRET_PF_SPURIOUSis not a valid return value.
Hmm, what are you referring to regarding RET_PF_SPURIOUS?
Quoted 7 lines
Then, what about setting
.max_level =PG_LEVEL_4K,
directly?Otherwise, the "(
KVM_BUG_ON(level != PG_LEVEL_4K, kvm)" would be triggered intdx_sept_set_private_spte().
Yes this fails to boot a TD. With max_level = PG_LEVEL_4K it passes the full tests. I don't think it's ideal to encode PAGE.ADD details here though.
But I'm not immediately clear what is going wrong. The old struct kvm_page_fault looks pretty similar. Did you root cause it?
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 9 lines · Rick Edgecombe
Quoted 8 lines
Then, what about setting
.max_level =PG_LEVEL_4K,
directly?Otherwise, the "(
KVM_BUG_ON(level != PG_LEVEL_4K, kvm)" would be triggered intdx_sept_set_private_spte().
Quoted 6 lines
Yes this fails to boot a TD. With
max_level=PG_LEVEL_4Kit passes the full tests. I don't think it's ideal to encode PAGE.ADD details here though.But I'm not immediately clear what is going wrong. The old struct
kvm_page_fault
looks pretty similar. Did you root cause it?
Oh, duh. Because we are passing in the PFN now so it can't know the size. So it's not about PAGE.ADD actually.
Sill, how about calling the function kvm_tdp_mmu_map_private_pfn_4k(), or passing in the level?
Re: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 13 lines · Rick P
Quoted 12 lines · Rick Edgecombe
Quoted 8 lines
Then, what about setting
.max_level =PG_LEVEL_4K,
directly?Otherwise, the "(
KVM_BUG_ON(level != PG_LEVEL_4K, kvm)" would be triggered intdx_sept_set_private_spte().Yes this fails to boot a TD. With
max_level=PG_LEVEL_4Kit passes the full tests. I don't think it's ideal to encode PAGE.ADD details here though.
Quoted 6 lines
Quoted 3 lines
But I'm not immediately clear what is going wrong. The old struct
kvm_page_fault
looks pretty similar. Did you root cause it?Oh, duh. Because we are passing in the PFN now so it can't know the size. So it's not about PAGE.ADD actually.
Right, it's because the previous kvm_tdp_map_page() updates fault->max_level in kvm_mmu_faultin_pfn_private() by checking the private_max_mapping_level hook.
However, private_max_mapping_level() skips the faultin step and goes straight to kvm_tdp_mmu_map().
Quoted 2 lines
Sill, how about calling the function
kvm_tdp_mmu_map_private_pfn_4k(), or passing in the level?
Looks [1] can also address this issue. Not sure which one Sean prefers.
Sean Christopherson
maintainerRe: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 23 lines · Yan Zhao
Quoted 13 lines · Rick P
Quoted 12 lines · Rick Edgecombe
Quoted 8 lines
Then, what about setting
.max_level =PG_LEVEL_4K,
directly?Otherwise, the "(
KVM_BUG_ON(level != PG_LEVEL_4K, kvm)" would be triggered intdx_sept_set_private_spte().Yes this fails to boot a TD. With
max_level=PG_LEVEL_4Kit passes the full tests. I don't think it's ideal to encode PAGE.ADD details here though.Quoted 6 lines
Quoted 3 lines
But I'm not immediately clear what is going wrong. The old struct
kvm_page_fault
looks pretty similar. Did you root cause it?Oh, duh. Because we are passing in the PFN now so it can't know the size. So it's not about PAGE.ADD actually.
Right, it's because the previous
kvm_tdp_map_page()updatesfault->max_levelinkvm_mmu_faultin_pfn_private()by checking theprivate_max_mapping_levelhook.
Quoted 8 lines
However,
private_max_mapping_level()skips the faultin step and goes straight tokvm_tdp_mmu_map().Quoted 2 lines
Sill, how about calling the function
kvm_tdp_mmu_map_private_pfn_4k(), or passing in the level?Looks [1] can also address this issue. Not sure which one Sean prefers.
That won't fix this issue though, becuase @fault will be valid and so max_level will still be KVM_MAX_HUGEPAGE_LEVEL. Which is by design, the intent in that flow is that KVM should have gotten the level when getting the pfn from gmem.
IIUC, this particular flow _must_ map at 4KiB, so I think forcing PG_LEVEL_4k is the right solution.
Re: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Yan Zhao
Quoted 13 lines · Rick P
Quoted 12 lines · Rick Edgecombe
Quoted 8 lines
Then, what about setting
.max_level =PG_LEVEL_4K,
directly?Otherwise, the "(
KVM_BUG_ON(level != PG_LEVEL_4K, kvm)" would be triggered intdx_sept_set_private_spte().Yes this fails to boot a TD. With
max_level=PG_LEVEL_4Kit passes the full tests. I don't think it's ideal to encode PAGE.ADD details here though.Quoted 6 lines
Quoted 3 lines
But I'm not immediately clear what is going wrong. The old struct
kvm_page_fault
looks pretty similar. Did you root cause it?Oh, duh. Because we are passing in the PFN now so it can't know the size. So it's not about PAGE.ADD actually.
Right, it's because the previous
kvm_tdp_map_page()updatesfault->max_levelinkvm_mmu_faultin_pfn_private()by checking theprivate_max_mapping_levelhook.However,
private_max_mapping_level()skips the faultin step and goes straight tokvm_tdp_mmu_map().
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Sill, how about calling the function
kvm_tdp_mmu_map_private_pfn_4k(), or passing in the level?Looks [1] can also address this issue. Not sure which one Sean prefers.
That won't fix this issue though, becuase @fault will be valid and so
max_level
Ah, right, I missed that you composed a fault...
Quoted 5 lines
will still be
KVM_MAX_HUGEPAGE_LEVEL. Which is by design, the intent in that flow is that KVM should have gotten the level when getting the pfn from gmem.IIUC, this particular flow _must_ map at 4KiB, so I think forcing
PG_LEVEL_4kis the right solution.
Forcing PG_LEVEL_4k looks good to me.
I was worried that SEV might want to use higher levels.
Re: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 28 lines · Yan Zhao
Quoted 27 lines · Sean Christopherson
Quoted 26 lines · Yan Zhao
Quoted 13 lines · Rick P
Quoted 12 lines · Rick Edgecombe
Quoted 8 lines
Then, what about setting
.max_level =PG_LEVEL_4K,
directly?Otherwise, the "(
KVM_BUG_ON(level != PG_LEVEL_4K, kvm)" would be triggered intdx_sept_set_private_spte().Yes this fails to boot a TD. With
max_level=PG_LEVEL_4Kit passes the full tests. I don't think it's ideal to encode PAGE.ADD details here though.Quoted 6 lines
Quoted 3 lines
But I'm not immediately clear what is going wrong. The old struct
kvm_page_fault
looks pretty similar. Did you root cause it?Oh, duh. Because we are passing in the PFN now so it can't know the size. So it's not about PAGE.ADD actually.
Right, it's because the previous
kvm_tdp_map_page()updatesfault->max_levelinkvm_mmu_faultin_pfn_private()by checking theprivate_max_mapping_levelhook.However,
private_max_mapping_level()skips the faultin step and goes straight tokvm_tdp_mmu_map().
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Sill, how about calling the function
kvm_tdp_mmu_map_private_pfn_4k(), or passing in the level?Looks [1] can also address this issue. Not sure which one Sean prefers.
That won't fix this issue though, becuase @fault will be valid and so
max_levelAh, right, I missed that you composed a fault...
FWIW: after reviewing it again, I think [1] is still able update the max_level to 4KB.
The flow with a valid @fault:
kvm_mmu_hugepage_adjust
kvm_mmu_max_mapping_level
kvm_max_private_mapping_level
kvm_x86_call(gmem_max_mapping_level)(kvm, pfn);Quoted 7 lines
Quoted 5 lines
will still be
KVM_MAX_HUGEPAGE_LEVEL. Which is by design, the intent in that flow is that KVM should have gotten the level when getting the pfn from gmem.IIUC, this particular flow _must_ map at 4KiB, so I think forcing
PG_LEVEL_4kis the right solution.Forcing
PG_LEVEL_4klooks good to me.
I was worried that SEV might want to use higher levels.
Re: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
In reply to Edgecombe, Rick PQuoted 15 lines · Rick P
Quoted 5 lines · Yan Zhao
Quoted 5 lines
arch/x86/kvm/mmu/mmu.c +4
-0+{ + struct kvm_page_fault fault = { + .addr = gfn_to_gpa(gfn), + .error_code = PFERR_GUEST_FINAL_MASK |
PFERR_PRIVATE_ACCESS,Quoted 8 lines
arch/x86/kvm/mmu/mmu.c long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · line 5002
Quoted 6 lines
arch/x86/kvm/mmu/mmu.c +3
-0+ .prefetch = true, + .is_tdp = true, + .nx_huge_page_workaround_enabled =
is_nx_huge_page_enabled(vcpu->kvm),arch/x86/kvm/mmu/mmu.c +2
-0+ + .max_level = KVM_MAX_HUGEPAGE_LEVEL,Looks the
kvm_tdp_mmu_map_private_pfn()is only for initial memory mapping, given that ".prefetch = true" andRET_PF_SPURIOUSis not a valid return value.
Quoted 1 line
Hmm, what are you referring to regarding
RET_PF_SPURIOUS?
If kvm_tdp_mmu_map_private_pfn() can also be invoked after initial memory mapping stage, RET_PF_SPURIOUS is a valid return case.
But in this patch, only RET_PF_RETRY and RET_PF_FIXED are valid. So, I think it's expected to be invoked only by initial memory mapping stage :)
Re: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
In reply to Sean ChristophersonSean Christopherson wrote:
[snip]
Quoted 16 lines
arch/x86/kvm/mmu/mmu.c +8
-0@@ -4990,6 +4990,65 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,return min (range->size, end - range->gpa); } +int kvm_tdp_mmu_map_private_pfn (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn) +{ +struct kvm_page_fault fault = { + .addr =gfn_to_gpa (gfn), + .error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS , + .prefetch = true, + .is_tdp = true, + .nx_huge_page_workaround_enabled =is_nx_huge_page_enabled (vcpu->kvm),
arch/x86/kvm/mmu/mmu.c long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · line 5010
Quoted 10 lines
arch/x86/kvm/mmu/mmu.c +10
-0+ + .max_level = KVM_MAX_HUGEPAGE_LEVEL, + .req_level = PG_LEVEL_4K, + .goal_level = PG_LEVEL_4K, + .is_private = true, + + .gfn = gfn, + .slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn), + .pfn = pfn, + .map_writable = true,
Why is map_writable set? Doesn't this get translated into host_writable?
Ira
[snip]
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 3 lines · Ira Weiny
arch/x86/kvm/mmu/mmu.c long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · line 5010
Quoted 1 line
arch/x86/kvm/mmu/mmu.c +1
-0+ .map_writable = true,Why is
map_writableset? Doesn't this get translated intohost_writable?
I guess it's normally set only if it's a !KVM_MEM_READONLY slot for private faults memory. But that flag is invalid for gmem. So we should only have map_writable=true cases for tdx.
Hypothetically this function should work for non-gmem. I guess since it's exported, a comment could be nice to specify that the memslots are not consulted. There are many MMU details that are not commented though, so it's probably too much given that the struct is right there to look at what kind of fault it is.
Sean Christopherson
maintainerRe: [RFC PATCH 02/12] KVM: x86/mmu: Add dedicated API to map guest_memfd pfn into TDP MMU
Quoted 7 lines · Rick P Edgecombe
Quoted 3 lines · Ira Weiny
arch/x86/kvm/mmu/mmu.c long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, · line 5010
Quoted 1 line
arch/x86/kvm/mmu/mmu.c +1
-0+ .map_writable = true,Why is
map_writableset? Doesn't this get translated intohost_writable?I guess it's normally set only if it's a !KVM_MEM_READONLY slot for private faults memory.
map_writable can also be %false on read faults and the host userspace mapping isn't writable.
Quoted 2 lines
But that flag is invalid for gmem. So we should only have map_writable=true cases for tdx.
Yep. And not TDX specific, map_writable _must_ be true for write faults. The reason there's two separate flags is so that KVM can opportunistically create a writable mapping on read faults.
Sean Christopherson
maintainerRFC PATCH 3/12 Revert "KVM: x86/tdp_mmu: Add a helper function to walk down the TDP MMU"
In reply to Sean ChristophersonRemove the helper and exports that were added to allow TDX code to reuse kvm_tdp_map_page() for its gmem post-populated flow now that a dedicated TDP MMU API is provided to install a mapping given a gfn+pfn pair.
This reverts commit 2608f105760115e94a03efd9f12f8fbfd1f9af4b.
--- arch/x86/kvm/mmu.h | 2 -- arch/x86/kvm/mmu/mmu.c | 4 ++-- arch/x86/kvm/mmu/tdp_mmu.c | 37 +++++-------------------------------- 3 files changed, 7 insertions(+), 36 deletions(-)
arch/x86/kvm/mmu.h +0 -2
@@ -257,8 +257,6 @@ extern bool tdp_mmu_enabled;#define tdp_mmu_enabled false#endif -bool kvm_tdp_mmu_gpa_is_mapped (struct kvm_vcpu *vcpu,u64 gpa); -int kvm_tdp_map_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code,u8 *level);int kvm_tdp_mmu_map_private_pfn (struct kvm_vcpu *vcpu,gfn_t gfn,kvm_pfn_t pfn);static inline bool kvm_memslots_have_rmaps (struct kvm *kvm)
arch/x86/kvm/mmu/mmu.c +2 -2
@@ -4900,7 +4900,8 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return direct_page_fault (vcpu, fault); } -int kvm_tdp_map_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code,u8 *level) +static int kvm_tdp_map_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code, +u8 *level) {int r;
@@ -4942,7 +4943,6 @@ int kvm_tdp_map_page(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 *levelreturn -EIO ; } } -EXPORT_SYMBOL_GPL (kvm_tdp_map_page);long kvm_arch_vcpu_pre_fault_memory (struct kvm_vcpu *vcpu,struct kvm_pre_fault_memory *range)
arch/x86/kvm/mmu/tdp_mmu.c +5 -32
@@ -1910,13 +1910,16 @@ bool kvm_tdp_mmu_write_protect_gfn(struct kvm *kvm,* * Must be called between kvm_tdp_mmu_walk_lockless_{begin,end}. */ -static int __kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes, -struct kvm_mmu_page *root) +int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes, +int *root_level) { +struct kvm_mmu_page *root =root_to_sp (vcpu->arch.mmu->root.hpa);struct tdp_iter iter;gfn_t gfn = addr >>PAGE_SHIFT ;int leaf =-1 ; + *root_level = vcpu->arch.mmu->root_role.level; +for_each_tdp_pte (iter, vcpu->kvm, root, gfn, gfn +1 ) { leaf = iter.level; sptes[leaf] = iter.old_spte;
@@ -1925,36 +1928,6 @@ static int __kvm_tdp_mmu_get_walk(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes,return leaf; } -int kvm_tdp_mmu_get_walk (struct kvm_vcpu *vcpu,u64 addr,u64 *sptes, -int *root_level) -{ -struct kvm_mmu_page *root =root_to_sp (vcpu->arch.mmu->root.hpa); - *root_level = vcpu->arch.mmu->root_role.level; - -return __kvm_tdp_mmu_get_walk (vcpu, addr, sptes, root); -} - -bool kvm_tdp_mmu_gpa_is_mapped (struct kvm_vcpu *vcpu,u64 gpa) -{ -struct kvm *kvm = vcpu->kvm; -bool is_direct =kvm_is_addr_direct (kvm, gpa); -hpa_t root = is_direct ? vcpu->arch.mmu->root.hpa : - vcpu->arch.mmu->mirror_root_hpa; -u64 sptes[PT64_ROOT_MAX_LEVEL +1 ], spte; -int leaf; - -lockdep_assert_held (&kvm->mmu_lock); -rcu_read_lock (); - leaf =__kvm_tdp_mmu_get_walk (vcpu, gpa, sptes,root_to_sp (root)); -rcu_read_unlock (); -if (leaf <0 ) -return false; - - spte = sptes[leaf]; -return is_shadow_present_pte (spte) &&is_last_spte (spte, leaf); -} -EXPORT_SYMBOL_GPL (kvm_tdp_mmu_gpa_is_mapped); -/* * Returns the last level spte pointer of the shadow page walk for the given * gpa, and sets *spte to the spte value. This spte may be non-preset. If no
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Sean Christopherson
maintainerRFC PATCH 4/12 KVM: x86/mmu: Rename kvm_tdp_map_page() to kvm_tdp_prefault_page()
In reply to Sean ChristophersonRename kvm_tdp_map_page() to kvm_tdp_prefault_page() now that it's used only by kvm_arch_vcpu_pre_fault_memory().
No functional change intended.
--- arch/x86/kvm/mmu/mmu.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +3 -3
@@ -4900,8 +4900,8 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)return direct_page_fault (vcpu, fault); } -static int kvm_tdp_map_page (struct kvm_vcpu *vcpu,gpa_t gpa,u64 error_code, -u8 *level) +static int kvm_tdp_prefault_page (struct kvm_vcpu *vcpu,gpa_t gpa, +u64 error_code,u8 *level) {int r;
@@ -4978,7 +4978,7 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu,* Shadow paging uses GVA for kvm page fault, so restrict to * two-dimensional paging. */ - r =kvm_tdp_map_page (vcpu, range->gpa | direct_bits, error_code, &level); + r =kvm_tdp_prefault_page (vcpu, range->gpa | direct_bits, error_code, &level);if (r <0 )return r;
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Edgecombe, Rick P
maintainerRe: [RFC PATCH 04/12] KVM: x86/mmu: Rename kvm_tdp_map_page() to kvm_tdp_prefault_page()
Quoted 4 lines · Sean Christopherson
Rename
kvm_tdp_map_page()tokvm_tdp_prefault_page()now that it's used only bykvm_arch_vcpu_pre_fault_memory().No functional change intended.
I realize you are just trying to do map->prefault here, but "page" seems redundant once you have "prefault" in the name. Why page here vs all the other fault handler functions without it?
Sean Christopherson
maintainerRe: [RFC PATCH 04/12] KVM: x86/mmu: Rename kvm_tdp_map_page() to kvm_tdp_prefault_page()
Quoted 9 lines · Rick P Edgecombe
Quoted 4 lines · Sean Christopherson
Rename
kvm_tdp_map_page()tokvm_tdp_prefault_page()now that it's used only bykvm_arch_vcpu_pre_fault_memory().No functional change intended.
I realize you are just trying to do
map->prefaulthere, but "page" seems redundant once you have "prefault" in the name. Why page here vs all the other fault handler functions without it?
kvm_tdp_prefault() feels a bit ambiguous/bare. Many of the fault helpers do have "page", it's just before the fault part.
kvm_mmu_finish_page_fault
kvm_handle_page_fault
kvm_tdp_page_fault
direct_page_fault
nonpaging_page_fault
kvm_tdp_mmu_page_fault
(and probably more)
How about kvm_tdp_page_prefault()? Or kvm_tdp_do_prefault(), but I think I like kvm_tdp_page_prefault() a little more.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 04/12] KVM: x86/mmu: Rename kvm_tdp_map_page() to kvm_tdp_prefault_page()
Quoted 8 lines · Sean Christopherson
Quoted 4 lines
I realize you are just trying to do
map->prefaulthere, but "page" seems redundant once you have "prefault" in the name. Why page here vs all the other fault handler functions without it?
kvm_tdp_prefault()feels a bit ambiguous/bare. Many of the fault helpers do have "page", it's just before the fault part.
Quoted 8 lines
kvm_mmu_finish_page_fault
kvm_handle_page_fault
kvm_tdp_page_fault
direct_page_fault
nonpaging_page_fault
kvm_tdp_mmu_page_fault(and probably more)
True.
Quoted 3 lines
How about kvm_tdp_page_prefault()? Or
kvm_tdp_do_prefault(), but I think I likekvm_tdp_page_prefault()a little more.
kvm_tdp_page_prefault() would be my pick of those.
Sean Christopherson
maintainerRFC PATCH 5/12 KVM: TDX: Drop superfluous page pinning in S-EPT management
In reply to Sean ChristophersonDon't explicitly pin pages when mapping pages into the S-EPT, guest_memfd doesn't support page migration in any capacity, i.e. there are no migrate callbacks because guest_memfd pages *can't* be migrated. See the WARN in kvm_gmem_migrate_folio().
--- arch/x86/kvm/vmx/tdx.c | 28 ++++------------------------ 1 file changed, 4 insertions(+), 24 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -24
@@ -1586,29 +1586,22 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) -{ -put_page (page); -} -static int tdx_mem_page_aug (struct kvm *kvm,gfn_t gfn, -enum pg_level level,struct page *page) +enum pg_level level,kvm_pfn_t pfn) {int tdx_level =pg_level_to_tdx_sept_level (level);struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +struct page *page =pfn_to_page (pfn);gpa_t gpa =gfn_to_gpa (gfn);u64 entry, level_state;u64 err; err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); -if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +if (unlikely (tdx_operand_busy (err)))return -EBUSY ; - }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page);return -EIO ; }
@@ -1642,29 +1635,18 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); -struct page *page =pfn_to_page (pfn);/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm))return -EINVAL ; -/* -* Because guest_memfd doesn't support page migration with -* a_ops->migrate_folio (yet), no callback is triggered for KVM on page -* migration. Until guest_memfd supports page migration, prevent page -* migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). -*/ -get_page (page); -/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching * barrier in tdx_td_finalize(). */ smp_rmb ();if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, page); +return tdx_mem_page_aug (kvm, gfn, level, pfn);return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); }
@@ -1715,7 +1697,6 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; }tdx_clear_page (page); -tdx_unpin (kvm, page);return 0 ; }
@@ -1795,7 +1776,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page);return 0 ; }
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 05/12] KVM: TDX: Drop superfluous page pinning in S-EPT management
Quoted 4 lines · Sean Christopherson
Don't explicitly pin pages when mapping pages into the S-EPT,
guest_memfddoesn't support page migration in any capacity, i.e. there are no migrate callbacks becauseguest_memfdpages *can't* be migrated. See the WARN inkvm_gmem_migrate_folio().
Hmm, we implemented exactly the same patch at [1], where we explained the potential problems of not holding page refcount, and the explored various approaches, and related considerations.
[1] https://lore.kernel.org/all/[email protected]/
Quoted 92 lines
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 28 ++++------------------------ 1 file changed, 4 insertions(+), 24 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-24@@ -1586,29 +1586,22 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) -{ -put_page (page); -} -static int tdx_mem_page_aug (struct kvm *kvm,gfn_t gfn, -enum pg_level level,struct page *page) +enum pg_level level,kvm_pfn_t pfn) {int tdx_level =pg_level_to_tdx_sept_level (level);struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +struct page *page =pfn_to_page (pfn);gpa_t gpa =gfn_to_gpa (gfn);u64 entry, level_state;u64 err; err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); -if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +if (unlikely (tdx_operand_busy (err)))return -EBUSY ; - }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page);return -EIO ; }@@ -1642,29 +1635,18 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); -struct page *page =pfn_to_page (pfn);/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm))return -EINVAL ; -/* -* Because guest_memfd doesn't support page migration with -* a_ops->migrate_folio (yet), no callback is triggered for KVM on page -* migration. Until guest_memfd supports page migration, prevent page -* migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). -*/ -get_page (page); -/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching * barrier in tdx_td_finalize(). */ smp_rmb ();if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, page); +return tdx_mem_page_aug (kvm, gfn, level, pfn);return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); }@@ -1715,7 +1697,6 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; }tdx_clear_page (page); -tdx_unpin (kvm, page);return 0 ; }@@ -1795,7 +1776,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page);return 0 ; }Signature · 1 line
2.51.0.268.g9569e192d0-goog
Edgecombe, Rick P
maintainerRe: [RFC PATCH 05/12] KVM: TDX: Drop superfluous page pinning in S-EPT management
Quoted 10 lines · Yan Zhao
Quoted 4 lines · Sean Christopherson
Don't explicitly pin pages when mapping pages into the S-EPT,
guest_memfddoesn't support page migration in any capacity, i.e. there are no migrate callbacks becauseguest_memfdpages *can't* be migrated. See the WARN inkvm_gmem_migrate_folio().Hmm, we implemented exactly the same patch at [1], where we explained the potential problems of not holding page refcount, and the explored various approaches, and related considerations.
Yea, so the outcome of the huge page related discussion was that we should look at some sort of emergency page reclaim feature for the TDX module to use in the case of bugs. But in the meantime to move forward without it, using a solution like in this patch.
Sean Christopherson
maintainerRe: [RFC PATCH 05/12] KVM: TDX: Drop superfluous page pinning in S-EPT management
Quoted 11 lines · Rick P Edgecombe
Quoted 10 lines · Yan Zhao
Quoted 4 lines · Sean Christopherson
Don't explicitly pin pages when mapping pages into the S-EPT,
guest_memfddoesn't support page migration in any capacity, i.e. there are no migrate callbacks becauseguest_memfdpages *can't* be migrated. See the WARN inkvm_gmem_migrate_folio().Hmm, we implemented exactly the same patch at [1], where we explained the potential problems of not holding page refcount, and the explored various approaches, and related considerations.
Oh, nice! I'll grab that and massage the changelog to break the hard dependencies on the rest of the hugepage support.
Quoted 4 lines
Yea, so the outcome of the huge page related discussion was that we should look at some sort of emergency page reclaim feature for the TDX module to use in the case of bugs. But in the meantime to move forward without it, using a solution like in this patch.
Re: [RFC PATCH 05/12] KVM: TDX: Drop superfluous page pinning in S-EPT management
In reply to Sean ChristophersonQuoted 4 lines · Sean Christopherson
Don't explicitly pin pages when mapping pages into the S-EPT,
guest_memfddoesn't support page migration in any capacity, i.e. there are no migrate callbacks becauseguest_memfdpages *can't* be migrated. See the WARN inkvm_gmem_migrate_folio().
I like the fact this removes a poorly named function tdx_unpin() as well.
That said, concerning gmem tracking page reference, I have some questions. In the TDX.PAGE.AUG path, [via kvm_gmem_get_pfn()] gmem takes a folio reference whereas the TDX.PAGE.ADD path [via kvm_gmem_populate()] does not take a folio reference.
Why are these paths different?
For this patch.
Quoted 93 lines
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 28 ++++------------------------ 1 file changed, 4 insertions(+), 24 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-24@@ -1586,29 +1586,22 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) -{ -put_page (page); -} -static int tdx_mem_page_aug (struct kvm *kvm,gfn_t gfn, -enum pg_level level,struct page *page) +enum pg_level level,kvm_pfn_t pfn) {int tdx_level =pg_level_to_tdx_sept_level (level);struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +struct page *page =pfn_to_page (pfn);gpa_t gpa =gfn_to_gpa (gfn);u64 entry, level_state;u64 err; err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); -if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +if (unlikely (tdx_operand_busy (err)))return -EBUSY ; - }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page);return -EIO ; }@@ -1642,29 +1635,18 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); -struct page *page =pfn_to_page (pfn);/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm))return -EINVAL ; -/* -* Because guest_memfd doesn't support page migration with -* a_ops->migrate_folio (yet), no callback is triggered for KVM on page -* migration. Until guest_memfd supports page migration, prevent page -* migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). -*/ -get_page (page); -/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching * barrier in tdx_td_finalize(). */ smp_rmb ();if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, page); +return tdx_mem_page_aug (kvm, gfn, level, pfn);return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); }@@ -1715,7 +1697,6 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; }tdx_clear_page (page); -tdx_unpin (kvm, page);return 0 ; }@@ -1795,7 +1776,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page);return 0 ; }Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 05/12] KVM: TDX: Drop superfluous page pinning in S-EPT management
Quoted 10 lines · Ira Weiny
Quoted 4 lines · Sean Christopherson
Don't explicitly pin pages when mapping pages into the S-EPT,
guest_memfddoesn't support page migration in any capacity, i.e. there are no migrate callbacks becauseguest_memfdpages *can't* be migrated. See the WARN inkvm_gmem_migrate_folio().I like the fact this removes a poorly named function
tdx_unpin()as well.That said, concerning gmem tracking page reference, I have some questions. In the TDX.PAGE.AUG path, [via
kvm_gmem_get_pfn()] gmem takes a folio
kvm_mmu_finish_page_fault() will decrease the folio refcount.
Quoted 104 lines
reference whereas the TDX.PAGE.ADD path [via
kvm_gmem_populate()] does not take a folio reference.Why are these paths different?
For this patch.
Reviewed-by Ira Weiny <[email protected]>Quoted 93 lines
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 28 ++++------------------------ 1 file changed, 4 insertions(+), 24 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-24@@ -1586,29 +1586,22 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) -{ -put_page (page); -} -static int tdx_mem_page_aug (struct kvm *kvm,gfn_t gfn, -enum pg_level level,struct page *page) +enum pg_level level,kvm_pfn_t pfn) {int tdx_level =pg_level_to_tdx_sept_level (level);struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +struct page *page =pfn_to_page (pfn);gpa_t gpa =gfn_to_gpa (gfn);u64 entry, level_state;u64 err; err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); -if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +if (unlikely (tdx_operand_busy (err)))return -EBUSY ; - }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page);return -EIO ; }@@ -1642,29 +1635,18 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, enum pg_level level, kvm_pfn_t pfn) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); -struct page *page =pfn_to_page (pfn);/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm))return -EINVAL ; -/* -* Because guest_memfd doesn't support page migration with -* a_ops->migrate_folio (yet), no callback is triggered for KVM on page -* migration. Until guest_memfd supports page migration, prevent page -* migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). -*/ -get_page (page); -/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching * barrier in tdx_td_finalize(). */ smp_rmb ();if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, page); +return tdx_mem_page_aug (kvm, gfn, level, pfn);return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); }@@ -1715,7 +1697,6 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; }tdx_clear_page (page); -tdx_unpin (kvm, page);return 0 ; }@@ -1795,7 +1776,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page);return 0 ; }Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 05/12] KVM: TDX: Drop superfluous page pinning in S-EPT management
Quoted 7 lines · Yan Zhao
Quoted 6 lines · Ira Weiny
Quoted 4 lines · Sean Christopherson
Don't explicitly pin pages when mapping pages into the S-EPT,
guest_memfddoesn't support page migration in any capacity, i.e. there are no migrate callbacks becauseguest_memfdpages *can't* be migrated. See the WARN inkvm_gmem_migrate_folio().
Quoted 5 lines
Quoted 4 lines
I like the fact this removes a poorly named function
tdx_unpin()as well.That said, concerning gmem tracking page reference, I have some questions. In the TDX.PAGE.AUG path, [via
kvm_gmem_get_pfn()] gmem takes a folio
kvm_mmu_finish_page_fault()will decrease the folio refcount.
Thanks,
Ira
Quoted 6 lines
Quoted 6 lines
reference whereas the TDX.PAGE.ADD path [via
kvm_gmem_populate()] does not take a folio reference.Why are these paths different?
[snip]
Re: [RFC PATCH 05/12] KVM: TDX: Drop superfluous page pinning in S-EPT management
In reply to Sean ChristophersonQuoted 7 lines · Sean Christopherson
Don't explicitly pin pages when mapping pages into the S-EPT,
guest_memfddoesn't support page migration in any capacity, i.e. there are no migrate callbacks becauseguest_memfdpages *can't* be migrated. See the WARN inkvm_gmem_migrate_folio().Signed-off-by Sean Christopherson <[email protected]>
Sean Christopherson
maintainerRFC PATCH 6/12 KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
In reply to Sean ChristophersonReturn -EIO when a KVM_BUG_ON() is tripped, as KVM's ABI is to return -EIO when a VM has been killed due to a KVM bug, not -EINVAL.
--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -4
@@ -1624,7 +1624,7 @@ static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, gfn_t gfn,struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EINVAL ; +return -EIO ;/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);
@@ -1638,7 +1638,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) -return -EINVAL ; +return -EIO ;/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching
@@ -1849,7 +1849,7 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,* and slot move/deletion. */ if (KVM_BUG_ON (is_hkid_assigned (kvm_tdx), kvm)) -return -EINVAL ; +return -EIO ;/* * The HKID assigned to this TD was already freed and cache was
@@ -1870,7 +1870,7 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,* there can't be anything populated in the private EPT. */ if (KVM_BUG_ON (!is_hkid_assigned (to_kvm_tdx (kvm)), kvm)) -return -EINVAL ; +return -EIO ; ret =tdx_sept_zap_private_spte (kvm, gfn, level, page);if (ret <=0 )
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
Quoted 2 lines · Sean Christopherson
Return
-EIOwhen aKVM_BUG_ON()is tripped, as KVM's ABI is to return-EIOwhen a VM has been killed due to a KVM bug, not-EINVAL.
Looks good to me, though currently the "-EIO" will not be returned to userspace either. In the fault path, RET_PF_RETRY is returned instead, while in the zap paths, void is returned.
Quoted 48 lines
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-4@@ -1624,7 +1624,7 @@ static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, gfn_t gfn,struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EINVAL ; +return -EIO ;/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);@@ -1638,7 +1638,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) -return -EINVAL ; +return -EIO ;/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching @@ -1849,7 +1849,7 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,* and slot move/deletion. */ if (KVM_BUG_ON (is_hkid_assigned (kvm_tdx), kvm)) -return -EINVAL ; +return -EIO ;/* * The HKID assigned to this TD was already freed and cache was @@ -1870,7 +1870,7 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,* there can't be anything populated in the private EPT. */ if (KVM_BUG_ON (!is_hkid_assigned (to_kvm_tdx (kvm)), kvm)) -return -EINVAL ; +return -EIO ; ret =tdx_sept_zap_private_spte (kvm, gfn, level, page);if (ret <=0 )Signature · 1 line
2.51.0.268.g9569e192d0-goog
Sean Christopherson
maintainerRe: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
Quoted 6 lines · Yan Zhao
Quoted 2 lines · Sean Christopherson
Return
-EIOwhen aKVM_BUG_ON()is tripped, as KVM's ABI is to return-EIOwhen a VM has been killed due to a KVM bug, not-EINVAL.Looks good to me, though currently the "
-EIO" will not be returned to userspace either. In the fault path,RET_PF_RETRYis returned instead, while in the zap paths, void is returned.
Yeah, I suspected as much. I'll call that out in the changeloge, i.e. that this is really just for internal consistency.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
In reply to Sean ChristophersonQuoted 40 lines · Sean Christopherson
Return
-EIOwhen aKVM_BUG_ON()is tripped, as KVM's ABI is to return-EIOwhen a VM has been killed due to a KVM bug, not-EINVAL.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/tdx.c +3
-3@@ -1624,7 +1624,7 @@ static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, gfn_t gfn,struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EINVAL ; +return -EIO ;/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);@@ -1638,7 +1638,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) -return -EINVAL ; +return -EIO ;/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching @@ -1849,7 +1849,7 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,* and slot move/deletion. */ if (KVM_BUG_ON (is_hkid_assigned (kvm_tdx), kvm)) -return -EINVAL ; +return -EIO ;/* * The HKID assigned to this TD was already freed and cache was
arch/x86/kvm/vmx/tdx.c static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, · line 1876
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +1
-1@@ -1870,7 +1870,7 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,* there can't be anything populated in the private EPT. */ if (KVM_BUG_ON (!is_hkid_assigned (to_kvm_tdx (kvm)), kvm)) -return -EINVAL ; +return -EIO ; ret =tdx_sept_zap_private_spte (kvm, gfn, level, page);if (ret <=0 )
Did you miss?
arch/x86/kvm/vmx/tdx.c +0 -0
@@ -1656,10 +1656,10 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm,
gfn_t gfn,
/* TODO: handle large pages. */
if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))- return -EINVAL;
+ return -EIO;
if (KVM_BUG_ON(!is_hkid_assigned(kvm_tdx), kvm))
- return -EINVAL;
+ return -EIO;
/*
* When zapping private page, write lock is held. So no race condition
We really have a lot of KVM_BUG_ON()s in tdx code. I hesitate to pull them out but it feels a bit gratuitous.
Sean Christopherson
maintainerRe: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
Quoted 47 lines · Rick P Edgecombe
Quoted 46 lines · Sean Christopherson
Return
-EIOwhen aKVM_BUG_ON()is tripped, as KVM's ABI is to return-EIOwhen a VM has been killed due to a KVM bug, not-EINVAL.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-4@@ -1624,7 +1624,7 @@ static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, gfn_t gfn,struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EINVAL ; +return -EIO ;/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);@@ -1638,7 +1638,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,/* TODO: handle large pages. */ if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) -return -EINVAL ; +return -EIO ;/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching @@ -1849,7 +1849,7 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,* and slot move/deletion. */ if (KVM_BUG_ON (is_hkid_assigned (kvm_tdx), kvm)) -return -EINVAL ; +return -EIO ;/* * The HKID assigned to this TD was already freed and cache was @@ -1870,7 +1870,7 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,* there can't be anything populated in the private EPT. */ if (KVM_BUG_ON (!is_hkid_assigned (to_kvm_tdx (kvm)), kvm)) -return -EINVAL ; +return -EIO ;
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn, · line 1876
Quoted 2 lines
arch/x86/kvm/vmx/tdx.c +0
-0ret =tdx_sept_zap_private_spte (kvm, gfn, level, page);if (ret <=0 )Did you miss?
I did indeed.
Quoted 20 lines
arch/x86/kvm/vmx/tdx.c +2
-2@@ -1656,10 +1656,10 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, /* TODO: handle large pages. */ if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm)) - return -EINVAL; + return -EIO;if (KVM_BUG_ON (!is_hkid_assigned (kvm_tdx), kvm)) -return -EINVAL ; +return -EIO ;/* * When zapping private page, write lock is held. So no race condition
Quoted 2 lines
We really have a lot of KVM_BUG_ON()s in tdx code. I hesitate to pull them out but it feels a bit gratuitous.
Generally speaking, the number of KVM_BUG_ON()s is fine. What we can do though is reduce the amount of boilerplate and the number of paths the propagate a SEAMCALL err through multiple layers, e.g. by eliminating single-use helpers (which is made easier by reducing boilerplate and thus lines of code).
Concretely, if we combine the KVM_BUG_ON() usage with pr_tdx_error():
#define __TDX_BUG_ON(__err, __fn_str, __kvm, __fmt, __args...) \
({ \
struct kvm *_kvm = (__kvm); \
bool __ret = !!(__err); \
\
if (WARN_ON_ONCE(__ret && (!_kvm || !_kvm->vm_bugged))) { \
if (_kvm) \
kvm_vm_bugged(_kvm); \
pr_err_ratelimited("SEAMCALL " __fn_str " failed: 0x%llx" \
__fmt "\n", __err, __args); \
} \
unlikely(__ret); \
})#define TDX_BUG_ON(__err, __fn, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, "%s", "")
#define TDX_BUG_ON_1(__err, __fn, __rcx, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx", __rcx)
#define TDX_BUG_ON_2(__err, __fn, __rcx, __rdx, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx", __rcx, __rdx)
#define TDX_BUG_ON_3(__err, __fn, __rcx, __rdx, __r8, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx, r8 0x%llx", __rcx, __rdx, __r8)
And a macro to handle retry when kicking vCPUs out of the guest:
#define tdh_do_no_vcpus(tdh_func, kvm, args...) \
({ \
struct kvm_tdx *__kvm_tdx = to_kvm_tdx(kvm); \
u64 __err; \
\
lockdep_assert_held_write(&kvm->mmu_lock); \
\
__err = tdh_func(args); \
if (unlikely(tdx_operand_busy(__err))) { \
WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, true); \
kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); \
\
__err = tdh_func(args); \
\
WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, false); \
} \
__err; \
})And do a bit of massaging, then we can end up e.g. this, which IMO is much easier to follow than the current form of tdx_sept_remove_private_spte(), which has several duplicate sanity checks and error handlers.
The tdh_do_no_vcpus() macro is a little mean, but I think it's a net positive as eliminates quite a lot of "noise", and thus makes it easier to focus on the logic. And alternative to a trampoline macro would be to implement a guard() and then do a scoped_guard(), but I think that'd be just as hard to read, and would require almost as much boilerplate as there is today.
static void tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
enum pg_level level, u64 spte)
{
struct page *page = pfn_to_page(spte_to_pfn(spte));
int tdx_level = pg_level_to_tdx_sept_level(level);
struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
gpa_t gpa = gfn_to_gpa(gfn);
u64 err, entry, level_state;
/*
* HKID is released after all private pages have been removed, and set
* before any might be populated. Warn if zapping is attempted when
* there can't be anything populated in the private EPT.
*/
if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm))
return;
/* TODO: handle large pages. */
if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
return;
err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa,
tdx_level, &entry, &level_state);
if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm))
return;
/*
* TDX requires TLB tracking before dropping private page. Do
* it here, although it is also done later.
*/
tdx_track(kvm);
/*
* When zapping private page, write lock is held. So no race condition
* with other vcpu sept operation.
* Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs.
*/
err = tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa,
tdx_level, &entry, &level_state);
if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm))
return;
err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm))
return;
tdx_clear_page(page);
}Edgecombe, Rick P
maintainerRe: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
Quoted 24 lines · Sean Christopherson
Generally speaking, the number of KVM_BUG_ON()s is fine. What we can do though is reduce the amount of boilerplate and the number of paths the propagate a SEAMCALL err through multiple layers, e.g. by eliminating single-use helpers (which is made easier by reducing boilerplate and thus lines of code).
Concretely, if we combine the
KVM_BUG_ON()usage withpr_tdx_error():#define
__TDX_BUG_ON(__err, __fn_str, __kvm, __fmt, __args...)\({ \ struct kvm *_kvm = (__kvm); \ bool __ret = !!(__err); \ \ if (WARN_ON_ONCE(__ret && (!_kvm || !_kvm->vm_bugged))) { \ if (_kvm) \ kvm_vm_bugged(_kvm); \ pr_err_ratelimited("SEAMCALL " __fn_str " failed: 0x%llx" \ __fmt "\n", __err, __args); \ } \ unlikely(__ret); \ })#define TDX_BUG_ON(__err, __fn, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, "%s", "")
Quoted 8 lines
#define TDX_BUG_ON_1(__err, __fn, __rcx, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx", __rcx)#define TDX_BUG_ON_2(__err, __fn, __rcx, __rdx, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx", __rcx, __rdx)#define TDX_BUG_ON_3(__err, __fn, __rcx, __rdx, __r8, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx, r8 0x%llx", __rcx, __rdx, __r8)
In general sounds good. But there it's a bit strange to specify them rcx, rdx, etc in a general helper. This is fallout from the existing chain of strange naming:
For example tdh_mem_range_block() plucks them from those registers and calls them ext_err1 due to their conditional meaning. Then KVM gives them some more meaning with 'entry' and 'level_state". Then prints them out as original register names. How about keeping the KVM names, like:
#define TDX_BUG_ON_2(__err, __fn, arg1, arg2, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #arg1 " 0x%llx, " #arg2 "
0x%llx", arg1, arg2)
so you get: entry: 0x00 level:0xF00
I *think* there is a way to make this work like var args and have a single function, but it becomes impossible for people to read.
Quoted 10 lines
And a macro to handle retry when kicking vCPUs out of the guest:
#define
tdh_do_no_vcpus(tdh_func, kvm, args...)\({ \ struct kvm_tdx *__kvm_tdx = to_kvm_tdx(kvm); \ u64 __err; \ \ lockdep_assert_held_write(&kvm->mmu_lock); \
There is a functional change in that the lock assert is not required if BUSY avoidance can be guaranteed to not happen. I don't think it should be needed today. I guess it's probably better to not rely on hitting rare races to catch an issue like that.
Quoted 63 lines
\ __err = tdh_func(args); \ if (unlikely(tdx_operand_busy(__err))) { \ WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, true); \ kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE); \ \ __err = tdh_func(args); \ \ WRITE_ONCE(__kvm_tdx->wait_for_sept_zap, false); \ } \ __err; \ })And do a bit of massaging, then we can end up e.g. this, which IMO is much easier to follow than the current form of
tdx_sept_remove_private_spte(), which has several duplicate sanity checks and error handlers.The
tdh_do_no_vcpus()macro is a little mean, but I think it's a net positive as eliminates quite a lot of "noise", and thus makes it easier to focus on the logic. And alternative to a trampoline macro would be to implement aguard()and then do ascoped_guard(), but I think that'd be just as hard to read, and would require almost as much boilerplate as there is today.static void tdx_sept_remove_private_spte(struct kvm *kvm,
gfn_tgfn,
enumpg_levellevel, u64 spte){ struct page *page = pfn_to_page(spte_to_pfn(spte)); int tdx_level = pg_level_to_tdx_sept_level(level); struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm); gpa_t gpa = gfn_to_gpa(gfn); u64 err, entry, level_state; /* * HKID is released after all private pages have been removed, and set * before any might be populated. Warn if zapping is attempted when * there can't be anything populated in the private EPT. */ if (KVM_BUG_ON(!is_hkid_assigned(to_kvm_tdx(kvm)), kvm)) return; /* TODO: handle large pages. */ if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm)) return; err = tdh_do_no_vcpus(tdh_mem_range_block, kvm, &kvm_tdx->td, gpa, tdx_level, &entry, &level_state); if (TDX_BUG_ON_2(err, TDH_MEM_RANGE_BLOCK, entry, level_state, kvm)) return; /* * TDX requires TLB tracking before dropping private page. Do * it here, although it is also done later. */ tdx_track(kvm); /* * When zapping private page, write lock is held. So no race condition * with other vcpu sept operation. * Race with TDH.VP.ENTER due to (0-step mitigation) and Guest TDCALLs. */ err = tdh_do_no_vcpus(tdh_mem_page_remove, kvm, &kvm_tdx->td, gpa, tdx_level, &entry, &level_state); if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_REMOVE, entry, level_state, kvm))
Quoted 8 lines
return; err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page); if (TDX_BUG_ON(err, TDH_PHYMEM_PAGE_WBINVD, kvm)) return; tdx_clear_page(page); }
Seems like tasteful macro-ization to me.
Sean Christopherson
maintainerRe: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
Quoted 43 lines · Rick P Edgecombe
Quoted 24 lines · Sean Christopherson
Generally speaking, the number of KVM_BUG_ON()s is fine. What we can do though is reduce the amount of boilerplate and the number of paths the propagate a SEAMCALL err through multiple layers, e.g. by eliminating single-use helpers (which is made easier by reducing boilerplate and thus lines of code).
Concretely, if we combine the
KVM_BUG_ON()usage withpr_tdx_error():#define
__TDX_BUG_ON(__err, __fn_str, __kvm, __fmt, __args...)\({ \ struct kvm *_kvm = (__kvm); \ bool __ret = !!(__err); \ \ if (WARN_ON_ONCE(__ret && (!_kvm || !_kvm->vm_bugged))) { \ if (_kvm) \ kvm_vm_bugged(_kvm); \ pr_err_ratelimited("SEAMCALL " __fn_str " failed: 0x%llx" \ __fmt "\n", __err, __args); \ } \ unlikely(__ret); \ })#define TDX_BUG_ON(__err, __fn, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, "%s", "")Quoted 8 lines
#define TDX_BUG_ON_1(__err, __fn, __rcx, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx", __rcx)#define TDX_BUG_ON_2(__err, __fn, __rcx, __rdx, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx", __rcx, __rdx)#define TDX_BUG_ON_3(__err, __fn, __rcx, __rdx, __r8, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", rcx 0x%llx, rdx 0x%llx, r8 0x%llx", __rcx, __rdx, __r8)In general sounds good. But there it's a bit strange to specify them rcx, rdx, etc in a general helper. This is fallout from the existing chain of strange naming:
For example
tdh_mem_range_block()plucks them from those registers and calls themext_err1due to their conditional meaning. Then KVM gives them some more meaning with 'entry' and 'level_state". Then prints them out as original register names. How about keeping the KVM names, like:
Quoted 5 lines
#define TDX_BUG_ON_2(__err, __fn, arg1, arg2, __kvm) \ __TDX_BUG_ON(__err, #__fn, __kvm, ", " #arg1 " 0x%llx, " #arg2 "0x%llx", arg1, arg2)
so you get: entry:
0x00level:0xF00
Ooh, nice, I'll tack on a patch.
Quoted 2 lines
I *think* there is a way to make this work like var args and have a single function, but it becomes impossible for people to read.
Heh, and would probably take two months to decipher the compiler errors in order to get it working :-)
Quoted 10 lines
Quoted 8 lines
And a macro to handle retry when kicking vCPUs out of the guest:
#define
tdh_do_no_vcpus(tdh_func, kvm, args...)\({ \ struct kvm_tdx *__kvm_tdx = to_kvm_tdx(kvm); \ u64 __err; \ \ lockdep_assert_held_write(&kvm->mmu_lock); \There is a functional change
Ugh, I missed that. I'll do a prep change to make that explicit.
Quoted 4 lines
in that the lock assert is not required if BUSY
avoidance can be guaranteed to not happen. I don't think it should be needed today. I guess it's probably better to not rely on hitting rare races to catch an issue like that.
But that's not actually what the code does. The lockdep assert won't trip because KVM never removes S-EPT entries under read-lock:
if (is_mirror_sp(sp)) {
KVM_BUG_ON(shared, kvm);
remove_external_spte(kvm, gfn, old_spte, level);
}Not because KVM actually guarantees -EBUSY is avoided. So the current code is flawed, it just doesn't cause problems.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
Quoted 2 lines · Sean Christopherson
But that's not actually what the code does. The lockdep assert won't trip because KVM never removes S-EPT entries under read-lock:
Right
Quoted 8 lines
if (is_mirror_sp(sp)) { KVM_BUG_ON(shared, kvm); remove_external_spte(kvm, gfn, old_spte, level); }Not because KVM actually guarantees
-EBUSYis avoided. So the current code is flawed, it just doesn't cause problems.
Flawed, as in the lockdep should assert regardless of EBUSY? Seems good to me. Probably if we wanted to try to call tdx_sept_remove_private_spte() under read lock with special plans to avoid EBUSY we should think twice anyway.
Sean Christopherson
maintainerRe: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
Quoted 12 lines · Rick P Edgecombe
Quoted 2 lines · Sean Christopherson
But that's not actually what the code does. The lockdep assert won't trip because KVM never removes S-EPT entries under read-lock:
Right
Quoted 6 lines
if (is_mirror_sp(sp)) { KVM_BUG_ON(shared, kvm); remove_external_spte(kvm, gfn, old_spte, level); }
Quoted 4 lines
Quoted 2 lines
Not because KVM actually guarantees
-EBUSYis avoided. So the current code is flawed, it just doesn't cause problems.Flawed, as in the lockdep should assert regardless of EBUSY?
Yep, exactly.
Quoted 3 lines
Seems good to me.
Probably if we wanted to try to calltdx_sept_remove_private_spte()under read lock with special plans to avoid EBUSY we should think twice anyway.
Heh, add a few zeros to "twice" :-D
Re: [RFC PATCH 06/12] KVM: TDX: Return -EIO, not -EINVAL, on a KVM_BUG_ON() condition
In reply to Sean ChristophersonQuoted 4 lines · Sean Christopherson
Return
-EIOwhen aKVM_BUG_ON()is tripped, as KVM's ABI is to return-EIOwhen a VM has been killed due to a KVM bug, not-EINVAL.Signed-off-by Sean Christopherson <[email protected]>
[snip]
Sean Christopherson
maintainerRFC PATCH 7/12 KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte()
In reply to Sean ChristophersonReturn -EIO immediately from tdx_sept_zap_private_spte() if the number of to-be-added pages underflows, so that the following "KVM_BUG_ON(err, kvm)" isn't also triggered. Isolating the check from the "is premap error" if-statement will also allow adding a lockdep assertion that premap errors are encountered if and only if slots_lock is held.
--- arch/x86/kvm/vmx/tdx.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -2
@@ -1773,8 +1773,10 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, err =tdh_mem_range_block (&kvm_tdx->td, gpa, tdx_level, &entry, &level_state);tdx_no_vcpus_enter_stop (kvm); } -if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && - !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) { +if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) { +if (KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) +return -EIO ; +atomic64_dec (&kvm_tdx->nr_premapped);return 0 ; }
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 07/12] KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte()
Quoted 8 lines · Sean Christopherson
Return
-EIOimmediately fromtdx_sept_zap_private_spte()if the number of to-be-added pages underflows, so that the following "KVM_BUG_ON(err, kvm)" isn't also triggered. Isolating the check from the "is premap error" if-statement will also allow adding a lockdep assertion that premap errors are encountered if and only ifslots_lockis held.Signed-off-by Sean Christopherson <[email protected]>---
Edgecombe, Rick P
maintainerRe: [RFC PATCH 07/12] KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte()
Quoted 11 lines · Rick Edgecombe
Quoted 8 lines · Sean Christopherson
Return
-EIOimmediately fromtdx_sept_zap_private_spte()if the number of to-be-added pages underflows, so that the following "KVM_BUG_ON(err, kvm)" isn't also triggered. Isolating the check from the "is premap error" if-statement will also allow adding a lockdep assertion that premap errors are encountered if and only ifslots_lockis held.Signed-off-by Sean Christopherson <[email protected]>---
Reviewed-by Rick Edgecombe <[email protected]>
There is actually another KVM_BUG_ON() in the path here:
static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
int level)
{
kvm_pfn_t old_pfn = spte_to_pfn(old_spte);
int ret;
/*
* External (TDX) SPTEs are limited to PG_LEVEL_4K, and external
* PTs are removed in a special order, involving free_external_spt().
* But remove_external_spte() will be called on non-leaf PTEs via
* __tdp_mmu_zap_root(), so avoid the error the former would return
* in this case.
*/
if (!is_last_spte(old_spte, level))
return;
/* Zapping leaf spte is allowed only when write lock is held. */
lockdep_assert_held_write(&kvm->mmu_lock);
/* Because write lock is held, operation should success. */
ret = kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_pfn);-> KVM_BUG_ON(ret, kvm);
We don't need to do it in this patch, but we could remove the return value in .remove_external_spte, and the KVM_BUG_ON(). Just let remove_external_spte handle it internally.
Re: [RFC PATCH 07/12] KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte()
Quoted 35 lines · Rick P
Quoted 11 lines · Rick Edgecombe
Quoted 8 lines · Sean Christopherson
Return
-EIOimmediately fromtdx_sept_zap_private_spte()if the number of to-be-added pages underflows, so that the following "KVM_BUG_ON(err, kvm)" isn't also triggered. Isolating the check from the "is premap error" if-statement will also allow adding a lockdep assertion that premap errors are encountered if and only ifslots_lockis held.Signed-off-by Sean Christopherson <[email protected]>---
Reviewed-by Rick Edgecombe <[email protected]>There is actually another
KVM_BUG_ON()in the path here:static void remove_external_spte(struct kvm *kvm,
gfn_tgfn, u64old_spte,
int level){ kvm_pfn_t old_pfn = spte_to_pfn(old_spte); int ret; /* * External (TDX) SPTEs are limited to PG_LEVEL_4K, and external * PTs are removed in a special order, involving free_external_spt(). * But remove_external_spte() will be called on non-leaf PTEs via * __tdp_mmu_zap_root(), so avoid the error the former would return * in this case. */ if (!is_last_spte(old_spte, level)) return; /* Zapping leaf spte is allowed only when write lock is held. */ lockdep_assert_held_write(&kvm->mmu_lock); /* Because write lock is held, operation should success. */ ret = kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_pfn);
Quoted 5 lines
->
KVM_BUG_ON(ret, kvm);We don't need to do it in this patch, but we could remove the return value in .remove_external_spte, and the
KVM_BUG_ON(). Just letremove_external_sptehandle it internally.
+1. Triggering KVM_BUG_ON() only in TDX internally is better.
Re: [RFC PATCH 07/12] KVM: TDX: Avoid a double-KVM_BUG_ON() in tdx_sept_zap_private_spte()
In reply to Sean ChristophersonQuoted 16 lines · Sean Christopherson
Return
-EIOimmediately fromtdx_sept_zap_private_spte()if the number of to-be-added pages underflows, so that the following "KVM_BUG_ON(err, kvm)" isn't also triggered. Isolating the check from the "is premap error" if-statement will also allow adding a lockdep assertion that premap errors are encountered if and only ifslots_lockis held.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)arch/x86/kvm/vmx/tdx.c +0
-0
arch/x86/kvm/vmx/tdx.c static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, · line 1778
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +3
-2@@ -1773,8 +1773,10 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, err =tdh_mem_range_block (&kvm_tdx->td, gpa, tdx_level, &entry, &level_state);tdx_no_vcpus_enter_stop (kvm); } -if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && - !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) { +if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) { +if (KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) +return -EIO ;
Won't this -EIO cause the KVM_BUG_ON on in remove_external_spte() to fire too?
static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
int level)
{
...
ret = kvm_x86_call(remove_external_spte)(kvm, gfn, level, old_pfn);
KVM_BUG_ON(ret, kvm);
}This patch is better than 3 bug ons but wouldn't it be better to make both KVM_BUG_ON's internal errors or debug?
Something like this:
arch/x86/kvm/vmx/tdx.c +4 -2
@@ -1774,14 +1774,16 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,tdx_no_vcpus_enter_stop (kvm); }if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) { -if (KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) +if (!atomic64_read (&kvm_tdx->nr_premapped)) { +pr_err ("nr_premapped underflow\n" );return -EIO ; + }atomic64_dec (&kvm_tdx->nr_premapped);return 0 ; } -if (KVM_BUG_ON (err, kvm)) { +if (err) {pr_tdx_error_2 (TDH_MEM_RANGE_BLOCK , err, entry, level_state);return -EIO ; }
Sean Christopherson
maintainerRFC PATCH 8/12 KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
In reply to Sean ChristophersonUse atomic64_dec_return() when decrementing the number of "pre-mapped" S-EPT pages to ensure that the count can't go negative without KVM noticing. In theory, checking for '0' and then decrementing in a separate operation could miss a 0=>-1 transition. In practice, such a condition is impossible because nr_premapped is protected by slots_lock, i.e. doesn't actually need to be an atomic (that wart will be addressed shortly).
Don't bother trying to keep the count non-negative, as the KVM_BUG_ON() ensures the VM is dead, i.e. there's no point in trying to limp along.
--- arch/x86/kvm/vmx/tdx.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
arch/x86/kvm/vmx/tdx.c +2 -4
@@ -1774,10 +1774,9 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,tdx_no_vcpus_enter_stop (kvm); }if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) { -if (KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) +if (KVM_BUG_ON (atomic64_dec_return (&kvm_tdx->nr_premapped) <0 , kvm))return -EIO ; -atomic64_dec (&kvm_tdx->nr_premapped);return 0 ; }
@@ -3162,8 +3161,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, goto out; } -if (!KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) -atomic64_dec (&kvm_tdx->nr_premapped); +KVM_BUG_ON (atomic64_dec_return (&kvm_tdx->nr_premapped) <0 , kvm);if (arg->flags &KVM_TDX_MEASURE_MEMORY_REGION ) {for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) {
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 08/12] KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
Quoted 7 lines · Sean Christopherson
Use
atomic64_dec_return()when decrementing the number of "pre-mapped" S-EPT pages to ensure that the count can't go negative without KVM noticing. In theory, checking for '0' and then decrementing in a separate operation could miss a 0=>-1 transition. In practice, such a condition is impossible becausenr_premappedis protected byslots_lock, i.e. doesn't actually need to be an atomic (that wart will be addressed shortly).
Quoted 5 lines
Don't bother trying to keep the count non-negative, as the
KVM_BUG_ON()ensures the VM is dead, i.e. there's no point in trying to limp along.Signed-off-by Sean Christopherson <[email protected]>---
This area has gone through a lot of designs. In the v19 era PAGE.ADD got performed deep inside the fault by stuffing the source page in the vCPU. Then we switched to having userspace call KVM_PRE_FAULT_MEMORY manually to pre-populare the mirror EPT, and then have TDX code look up the PFN. Then nearer the end, we switched to current code which does something like KVM_PRE_FAULT_MEMORY internally, then looks up what got faulted and does the PAGE.ADD. Then the version in this series which does it even more directly.
nr_premapped got added during the KVM_PRE_FAULT_MEMORY era. I personally didn't like it, but it was needed because userspace could do unexpected things. Now it seems like its only purpose is to generate a KVM_BUG_ON() in tdx_sept_zap_private_spte(). I wonder if we could drop it all together and accept less KVM_BUG_ON() coverage. It seems weird to focus in on this specific error case.
Yan, am I missing something?
Re: [RFC PATCH 08/12] KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
Quoted 16 lines · Rick P
Quoted 7 lines · Sean Christopherson
Use
atomic64_dec_return()when decrementing the number of "pre-mapped" S-EPT pages to ensure that the count can't go negative without KVM noticing. In theory, checking for '0' and then decrementing in a separate operation could miss a 0=>-1 transition. In practice, such a condition is impossible becausenr_premappedis protected byslots_lock, i.e. doesn't actually need to be an atomic (that wart will be addressed shortly).Quoted 5 lines
Don't bother trying to keep the count non-negative, as the
KVM_BUG_ON()ensures the VM is dead, i.e. there's no point in trying to limp along.Signed-off-by Sean Christopherson <[email protected]>---
Reviewed-by Rick Edgecombe <[email protected]>
Quoted 7 lines
This area has gone through a lot of designs. In the v19 era PAGE.ADD got performed deep inside the fault by stuffing the source page in the vCPU. Then we switched to having userspace call
KVM_PRE_FAULT_MEMORYmanually to pre-populare the mirror EPT, and then have TDX code look up the PFN. Then nearer the end, we switched to current code which does something likeKVM_PRE_FAULT_MEMORYinternally, then looks up what got faulted and does the PAGE.ADD. Then the version in this series which does it even more directly.
Right.
If we invoke PAGE.ADD directly in tdx_sept_set_private_spte() (similar to PAGE.AUG), then we'll have to have some way to pass in the source page info.
So, rather than passing around the source page, we opted to record the count of pages mapped in M-EPT while still unmapped in S-EPT, i.e.,
1. map a page in M-EPT
2. increase nr_premapped.
3. map the page in S-EPT
4. decrease nr_premapped.
If a page is zapped in M-EPT before 3, decrease nr_premapped. So if 3 is executed successfully after zapping of the M-EPT, decrease nr_premapped too. The unbalancing of nr_premapped due to the double decrease indicates the mismatching between M-EPT and S-EPT.
If 3 never comes or fails, it's ok.
Quoted 8 lines
nr_premappedgot added during theKVM_PRE_FAULT_MEMORYera. I personally didn't like it, but it was needed because userspace could do unexpected things. Now it seems like its only purpose is to generate aKVM_BUG_ON()intdx_sept_zap_private_spte(). I wonder if we could drop it all together and accept lessKVM_BUG_ON()coverage. It seems weird to focus in on this specific error case.Yan, am I missing something?
Hmm, I still think it's safer to keep the nr_premapped to detect any unexpected code change.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/12] KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
Quoted 2 lines · Yan Zhao
Hmm, I still think it's safer to keep the
nr_premappedto detect any unexpected code change.
When I checking patch 6 I saw how many more KVM_BUG_ON()s we ended up with in TDX code compared to the rest of KVM. (even after we dropped a bunch during development) We have to differentiate from good safety, and "safety" that is really just propping up brittle code. Each KVM_BUG_ON() is a hint that there might be design issues.
Sean Christopherson
maintainerRe: [RFC PATCH 08/12] KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
Quoted 9 lines · Rick P Edgecombe
Quoted 2 lines · Yan Zhao
Hmm, I still think it's safer to keep the
nr_premappedto detect any unexpected code change.When I checking patch 6 I saw how many more KVM_BUG_ON()s we ended up with in TDX code compared to the rest of KVM. (even after we dropped a bunch during development) We have to differentiate from good safety, and "safety" that is really just propping up brittle code. Each
KVM_BUG_ON()is a hint that there might be design issues.
Nah, I think we're good. The majority of the asserts are on SEAMCALLs, and those are no different than the WARN_ONCE() in vmx_insn_failed(), just spread out to individual call sites.
Once those are out of the numbers are entirely reasonable (WARNs and KVM_BUG_ON are both assertions against bugs, one is just guaranteed to be fatal to the VM).
$ git grep -e KVM_BUG_ON -e WARN_ vmx/tdx.c | wc -l 25 $ git grep -e KVM_BUG_ON -e WARN_ | wc -l 459
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/12] KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
Quoted 15 lines · Sean Christopherson
Quoted 9 lines · Rick P Edgecombe
Quoted 2 lines · Yan Zhao
Hmm, I still think it's safer to keep the
nr_premappedto detect any unexpected code change.When I checking patch 6 I saw how many more KVM_BUG_ON()s we ended up with in TDX code compared to the rest of KVM. (even after we dropped a bunch during development) We have to differentiate from good safety, and "safety" that is really just propping up brittle code. Each
KVM_BUG_ON()is a hint that there might be design issues.Nah, I think we're good. The majority of the asserts are on SEAMCALLs, and those are no different than the
WARN_ONCE()invmx_insn_failed(), just spread out to individual call sites.
Quoted 7 lines
Once those are out of the numbers are entirely reasonable (WARNs and
KVM_BUG_ONare both assertions against bugs, one is just guaranteed to be fatal to the VM).$ git grep -e KVM_BUG_ON -e WARN_ vmx/tdx.c | wc -l 25 $ git grep -e KVM_BUG_ON -e WARN_ | wc -l 459
Hmm, ok.
Re: [RFC PATCH 08/12] KVM: TDX: Use atomic64_dec_return() instead of a poor equivalent
In reply to Sean ChristophersonQuoted 11 lines · Sean Christopherson
Use
atomic64_dec_return()when decrementing the number of "pre-mapped" S-EPT pages to ensure that the count can't go negative without KVM noticing. In theory, checking for '0' and then decrementing in a separate operation could miss a 0=>-1 transition. In practice, such a condition is impossible becausenr_premappedis protected byslots_lock, i.e. doesn't actually need to be an atomic (that wart will be addressed shortly).Don't bother trying to keep the count non-negative, as the
KVM_BUG_ON()ensures the VM is dead, i.e. there's no point in trying to limp along.Signed-off-by Sean Christopherson <[email protected]>
[snip]
Sean Christopherson
maintainerRFC PATCH 9/12 KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Sean ChristophersonFold tdx_mem_page_record_premap_cnt() into tdx_sept_set_private_spte() as providing a one-off helper for effectively three lines of code is at best a wash, and splitting the code makes the comment for smp_rmb() _extremely_ confusing as the comment talks about reading kvm->arch.pre_fault_allowed before kvm_tdx->state, but the immediately visible code does the exact opposite.
Opportunistically rewrite the comments to more explicitly explain who is checking what, as well as _why_ the ordering matters.
No functional change intended.
--- arch/x86/kvm/vmx/tdx.c | 49 ++++++++++++++++++------------------------ 1 file changed, 21 insertions(+), 28 deletions(-)
arch/x86/kvm/vmx/tdx.c +21 -28
@@ -1608,29 +1608,6 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,return 0 ; } -/* -* KVM_TDX_INIT_MEM_REGION calls kvm_gmem_populate() to map guest pages; the -* callback tdx_gmem_post_populate() then maps pages into private memory. -* through the a seamcall TDH.MEM.PAGE.ADD(). The SEAMCALL also requires the -* private EPT structures for the page to have been built before, which is -* done via kvm_tdp_map_page(). nr_premapped counts the number of pages that -* were added to the EPT structures but not added with TDH.MEM.PAGE.ADD(). -* The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there -* are no half-initialized shared EPT pages. -*/ -static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn) -{ -struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); - -if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EIO ; - -/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ -atomic64_inc (&kvm_tdx->nr_premapped); -return 0 ; -} -static int tdx_sept_set_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn) {
@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;/* -* Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching -* barrier in tdx_td_finalize(). +* Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() +* before kvm_tdx->state. Userspace must not be allowed to pre-fault +* arbitrary memory until the initial memory image is finalized. Pairs +* with the smp_wmb() in tdx_td_finalize(). */ smp_rmb (); -if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, pfn); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +/* +* If the TD isn't finalized/runnable, then userspace is initializing +* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of +* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD +* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks +* the counter to ensure all mapped pages have been added to the image, +* to prevent running the TD with uninitialized memory. +*/ +if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) { +if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) +return -EIO ; + +atomic64_inc (&kvm_tdx->nr_premapped); +return 0 ; + } + +return tdx_mem_page_aug (kvm, gfn, level, pfn); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 64 lines · Sean Christopherson
Fold
tdx_mem_page_record_premap_cnt()intotdx_sept_set_private_spte()as providing a one-off helper for effectively three lines of code is at best a wash, and splitting the code makes the comment forsmp_rmb()_extremely_ confusing as the comment talks about readingkvm->arch.pre_fault_allowedbeforekvm_tdx->state, but the immediately visible code does the exact opposite.Opportunistically rewrite the comments to more explicitly explain who is checking what, as well as _why_ the ordering matters.
No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 49 ++++++++++++++++++------------------------ 1 file changed, 21 insertions(+), 28 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-26@@ -1608,29 +1608,6 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,return 0 ; } -/* -* KVM_TDX_INIT_MEM_REGION calls kvm_gmem_populate() to map guest pages; the -* callback tdx_gmem_post_populate() then maps pages into private memory. -* through the a seamcall TDH.MEM.PAGE.ADD(). The SEAMCALL also requires the -* private EPT structures for the page to have been built before, which is -* done via kvm_tdp_map_page(). nr_premapped counts the number of pages that -* were added to the EPT structures but not added with TDH.MEM.PAGE.ADD(). -* The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there -* are no half-initialized shared EPT pages. -*/ -static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn) -{ -struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); - -if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EIO ; - -/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ -atomic64_inc (&kvm_tdx->nr_premapped); -return 0 ; -} -static int tdx_sept_set_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn) {@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;/* -* Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching -* barrier in tdx_td_finalize(). +* Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() +* before kvm_tdx->state. Userspace must not be allowed to pre-fault +* arbitrary memory until the initial memory image is finalized. Pairs +* with the smp_wmb() in tdx_td_finalize(). */ smp_rmb (); -if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE ))
arch/x86/kvm/vmx/tdx.c static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1634
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-2-return tdx_mem_page_aug (kvm, gfn, level, pfn); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +/* +* If the TD isn't finalized/runnable, then userspace is initializing +* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of +* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD +* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks +* the counter to ensure all mapped pages have been added to the image, +* to prevent running the TD with uninitialized memory.
To prevent the mismatch between mirror EPT and the S-EPT?
e.g., Before KVM_TDX_FINALIZE_VM,
if userspace performs a zap after the TDH.MEM.PAGE.ADD, the page will be removed from the S-EPT. The count of nr_premapped will not change after the successful TDH.MEM.RANGE.BLOCK and TDH.MEM.PAGE.REMOVE.
As a result, the TD will still run with uninitialized memory.
Quoted 16 lines
arch/x86/kvm/vmx/tdx.c +10
-0+*/ +if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) { +if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) +return -EIO ; + +atomic64_inc (&kvm_tdx->nr_premapped); +return 0 ; + } + +return tdx_mem_page_aug (kvm, gfn, level, pfn); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,Signature · 1 line
2.51.0.268.g9569e192d0-goog
Sean Christopherson
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 16 lines · Yan Zhao
Quoted 15 lines · Sean Christopherson
arch/x86/kvm/vmx/tdx.c +4
-4@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;/* -* Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching -* barrier in tdx_td_finalize(). +* Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() +* before kvm_tdx->state. Userspace must not be allowed to pre-fault +* arbitrary memory until the initial memory image is finalized. Pairs +* with the smp_wmb() in tdx_td_finalize(). */ smp_rmb (); -if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, pfn);
Quoted 9 lines
arch/x86/kvm/vmx/tdx.c static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1634
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +7
-1-return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +/* +* If the TD isn't finalized/runnable, then userspace is initializing +* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of +* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD +* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks +* the counter to ensure all mapped pages have been added to the image, +* to prevent running the TD with uninitialized memory. To prevent the mismatch between mirror EPT and the S-EPT?
No? Because KVM bumps the count when installing the S-EPT and decrements it on AUG, so I don't see how nr_premapped guards against M-EPT vs. S-EPT issues?
Quoted 4 lines
e.g., Before
KVM_TDX_FINALIZE_VM, if userspace performs a zap after the TDH.MEM.PAGE.ADD, the page will be removed from the S-EPT. The count ofnr_premappedwill not change after the successful TDH.MEM.RANGE.BLOCK and TDH.MEM.PAGE.REMOVE.
Eww. It would be nice to close that hole, but I suppose it's futile, e.g. the underlying problem is unexpectedly removing pages from the initial, whether the VMM is doing stupid things before vs. after FINALIZE doesn't really matter.
Quoted 1 line
As a result, the TD will still run with uninitialized memory.
No? Because BLOCK+REMOVE means there are no valid S-EPT mappings. There's a "hole" that the guest might not expect, but that hole will trigger an EPT violation and only get "filled" if the guest explicitly accepts an AUG'd page.
Side topic, why does KVM tolerate tdh_mem_page_add() failure? IIUC, playing nice with tdh_mem_page_add() failure necessitates both the tdx_is_sept_zap_err_due_to_premap() craziness and the check in tdx_td_finalize() that all pending pages have been consumed.
What reasonable use case is there for gracefully handling tdh_mem_page_add() failure?
If there is a need to handle failure, I gotta imagine it's only for the -EBUSY case. And if it's only for -EBUSY, why can't that be handled by retrying in tdx_vcpu_init_mem_region()? If tdx_vcpu_init_mem_region() guarantees that all pages mapped into the S-EPT are ADDed, then it can assert that there are no pending pages when it completes (even if it "fails"), and similarly tdx_td_finalize() can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
Quoted 16 lines
Quoted 16 lines
arch/x86/kvm/vmx/tdx.c +10
-0+*/ +if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) { +if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) +return -EIO ; + +atomic64_inc (&kvm_tdx->nr_premapped); +return 0 ; + } + +return tdx_mem_page_aug (kvm, gfn, level, pfn); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,Signature · 1 line
2.51.0.268.g9569e192d0-goog
Edgecombe, Rick P
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 10 lines · Sean Christopherson
Quoted 4 lines
e.g., Before
KVM_TDX_FINALIZE_VM, if userspace performs a zap after the TDH.MEM.PAGE.ADD, the page will be removed from the S-EPT. The count ofnr_premappedwill not change after the successful TDH.MEM.RANGE.BLOCK and TDH.MEM.PAGE.REMOVE.Eww. It would be nice to close that hole, but I suppose it's futile, e.g. the underlying problem is unexpectedly removing pages from the initial, whether the
VMM is doing stupid things before vs. after FINALIZE doesn't really matter.
Quoted 5 lines
Quoted 1 line
As a result, the TD will still run with uninitialized memory.
No? Because BLOCK+REMOVE means there are no valid S-EPT mappings. There's a "hole" that the guest might not expect, but that hole will trigger an EPT violation and only get "filled" if the guest explicitly accepts an AUG'd page.
Ah, I just responded on another patch. I wonder if we can get rid of the premap cnt.
Quoted 6 lines
Side topic, why does KVM tolerate
tdh_mem_page_add()failure? IIUC, playing nice withtdh_mem_page_add()failure necessitates both thetdx_is_sept_zap_err_due_to_premap()craziness and the check intdx_td_finalize()
that all pending pages have been consumed.
Reasons that tdh_mem_page_add() could get BUSY:
1. If two vCPU's tried to tdh_mem_page_add() the same gpa at the same time they could contend the SEPT entry lock
2. If one vCPU tries to tdh_mem_page_add() while the other zaps (i.e. tdh_mem_range_block()).
I guess since we don't hold MMU lock while we tdh_mem_page_add(), 2 is a possibility.
Quoted 11 lines
What reasonable use case is there for gracefully handling
tdh_mem_page_add()failure?If there is a need to handle failure, I gotta imagine it's only for the
-EBUSYcase. And if it's only for-EBUSY, why can't that be handled by retrying in tdx_vcpu_init_mem_region()? Iftdx_vcpu_init_mem_region()guarantees that all pages mapped into the S-EPT are ADDed, then it can assert that there are no pending pages when it completes (even if it "fails"), and similarlytdx_td_finalize()can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
Maybe we could take mmu write lock for the retry of tdh_mem_page_add(). Or maybe even for a single call of it, until someone wants to parallelize the operation.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 11 lines · Rick P
Quoted 10 lines · Sean Christopherson
Quoted 4 lines
e.g., Before
KVM_TDX_FINALIZE_VM, if userspace performs a zap after the TDH.MEM.PAGE.ADD, the page will be removed from the S-EPT. The count ofnr_premappedwill not change after the successful TDH.MEM.RANGE.BLOCK and TDH.MEM.PAGE.REMOVE.Eww. It would be nice to close that hole, but I suppose it's futile, e.g. the underlying problem is unexpectedly removing pages from the initial, whether the
VMM is doing stupid things before vs. after FINALIZE doesn't really matter.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
As a result, the TD will still run with uninitialized memory.
No? Because BLOCK+REMOVE means there are no valid S-EPT mappings. There's a "hole" that the guest might not expect, but that hole will trigger an EPT violation and only get "filled" if the guest explicitly accepts an AUG'd page.
Ah, I just responded on another patch. I wonder if we can get rid of the premap cnt.
I think keeping it is safer.
See my explanation at [1].
[1] https://lore.kernel.org/all/aK%[email protected].
Quoted 7 lines
Quoted 6 lines
Side topic, why does KVM tolerate
tdh_mem_page_add()failure? IIUC, playing nice withtdh_mem_page_add()failure necessitates both thetdx_is_sept_zap_err_due_to_premap()craziness and the check intdx_td_finalize()
that all pending pages have been consumed.
Quoted 5 lines
Reasons that
tdh_mem_page_add()could get BUSY:
1. If two vCPU's tried totdh_mem_page_add()the same gpa at the same time they could contend the SEPT entry lock
2. If one vCPU tries totdh_mem_page_add()while the other zaps (i.e.tdh_mem_range_block()).
Hmm, two tdh_mem_page_add()s can't contend as they are protected by both slot_lock and filemap lock.
With regard to the contention to tdh_mem_range_block(), please check my analysis at the above [1].
tdh_mem_page_add() could get BUSY though, when a misbehaved userspace invokes KVM_TDX_INIT_MEM_REGION on one vCPU while initializing another vCPU.
Please check more details at [2].
[2] https://lore.kernel.org/kvm/[email protected]/
Quoted 2 lines
I guess since we don't hold MMU lock while we
tdh_mem_page_add(), 2 is a possibility.
2 is possible only for paranoid zaps.
See "case 3. Unexpected zaps" in [1].
Quoted 11 lines
Quoted 10 lines
What reasonable use case is there for gracefully handling
tdh_mem_page_add()failure?If there is a need to handle failure, I gotta imagine it's only for the
-EBUSYcase. And if it's only for-EBUSY, why can't that be handled by retrying in tdx_vcpu_init_mem_region()? Iftdx_vcpu_init_mem_region()guarantees that all pages mapped into the S-EPT are ADDed, then it can assert that there are no pending pages when it completes (even if it "fails"), and similarlytdx_td_finalize()can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
Quoted 2 lines
Maybe we could take mmu write lock for the retry of
tdh_mem_page_add(). Or maybe even for a single call of it, until someone wants to parallelize the operation.
Hmm. I prefer returning -BUSY directly as invoking KVM_TDX_INIT_MEM_REGION before finishing initializing all vCPUs are uncommon.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 11 lines · Yan Zhao
Quoted 6 lines
Reasons that
tdh_mem_page_add()could get BUSY:
1. If two vCPU's tried totdh_mem_page_add()the same gpa at the same time they
could contend the SEPT entry lock
2. If one vCPU tries totdh_mem_page_add()while the other zaps (i.e.tdh_mem_range_block()).Hmm, two tdh_mem_page_add()s can't contend as they are protected by both
slot_lockand filemap lock.With regard to the contention to
tdh_mem_range_block(), please check my analysis at the above [1].
The analysis missed the tdh_mem_page_add() failure path
Quoted 7 lines
tdh_mem_page_add()could get BUSY though, when a misbehaved userspace invokesKVM_TDX_INIT_MEM_REGIONon one vCPU while initializing another vCPU.Please check more details at [2].
Ah, the TDR lock. I actually referred to an older version of your locking analysis that didn't have that one. But this means the premap count could get out of sync for that reason too.
Quoted 6 lines
Quoted 4 lines
I guess since we don't hold MMU lock while we
tdh_mem_page_add(), 2 is a possibility.2 is possible only for paranoid zaps.
See "case 3. Unexpected zaps" in [1].
Sean's lockdep assert handles half of those cases. Maybe we could also re- consider a KVM_BUG_ON() in the invalid zap paths again if it comes to it.
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
What reasonable use case is there for gracefully handling
tdh_mem_page_add()
failure?If there is a need to handle failure, I gotta imagine it's only for the - EBUSY
case. And if it's only for-EBUSY, why can't that be handled by retrying in
tdx_vcpu_init_mem_region()? Iftdx_vcpu_init_mem_region()guarantees that all
pages mapped into the S-EPT are ADDed, then it can assert that there are no
pending pages when it completes (even if it "fails"), and similarlytdx_td_finalize()can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
Quoted 6 lines
Quoted 4 lines
Maybe we could take mmu write lock for the retry of
tdh_mem_page_add(). Or maybe
even for a single call of it, until someone wants to parallelize the operation.Hmm. I prefer returning -BUSY directly as invoking
KVM_TDX_INIT_MEM_REGIONbefore finishing initializing all vCPUs are uncommon.
I was looking guaranteeing its success when Sean posted his suggestion to return to the original pattern. I'm in favor of that direction. If you agree we can call this moot.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Sean ChristophersonQuoted 27 lines · Sean Christopherson
Quoted 16 lines · Yan Zhao
Quoted 15 lines · Sean Christopherson
arch/x86/kvm/vmx/tdx.c +4
-4@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;/* -* Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching -* barrier in tdx_td_finalize(). +* Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() +* before kvm_tdx->state. Userspace must not be allowed to pre-fault +* arbitrary memory until the initial memory image is finalized. Pairs +* with the smp_wmb() in tdx_td_finalize(). */ smp_rmb (); -if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, pfn);Quoted 9 lines
arch/x86/kvm/vmx/tdx.c static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1634
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +7
-1-return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +/* +* If the TD isn't finalized/runnable, then userspace is initializing +* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of +* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD +* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks +* the counter to ensure all mapped pages have been added to the image, +* to prevent running the TD with uninitialized memory. To prevent the mismatch between mirror EPT and the S-EPT?
Quoted 2 lines
No? Because KVM bumps the count when installing the S-EPT and decrements it on AUG, so I don't see how
nr_premappedguards against M-EPT vs. S-EPT issues?
Hmm, I think there must be some misunderstanding.
Before userspace invokes KVM_TDX_FINALIZE_VM,
=======
1. the normal path (userspace invokes KVM_TDX_INIT_MEM_REGION).
(1) KVM holds slot_lock and filemap lock.
(2) KVM invokes kvm_tdp_map_page() (or kvm_tdp_mmu_map_private_pfn() in patch 2).
KVM increases nr_premapped in tdx_sept_set_private_spte() to indicate that there's a page mapped in M-EPT, while it's not yet installed in S-EPT.
(3) KVM invokes TDH.MEM.PAGE.ADD and decreases nr_premapped, indicating the page has been mapped in S-EPT too.
As the name of nr_premapped indicates, the count means a page is pre-mapped in the M-EPT, before its real mapping in the S-EPT.
If ADD fails in step (3), nr_premapped will not be decreased.
With mere the normal path, nr_premapped should return back to 0 after all KVM_TDX_INIT_MEM_REGIONs.
2. Expected zap paths (e.g. If userspace does something strange, such as removing a slot after KVM_TDX_INIT_MEM_REGION)
Those zap paths could be triggered by
1) userspace performs a page attribute conversion
2) userspace invokes gmem punch hole
3) userspace removes a slot
As all those paths either hold a slot_lock or a filemap lock, they can't contend with tdx_vcpu_init_mem_region() (tdx_vcpu_init_mem_region holds both slot_lock and internally filemap lock).
Consequently, those zaps must occur
a) before kvm_tdp_map_page() or
b) after TDH.MEM.PAGE.ADD.
For a), tdx_sept_zap_private_spte() won't not be invoked as the page is not
mapped in M-EPT either;
For b), tdx_sept_zap_private_spte() should succeed, as the BLOCK and REMOVE
SEAMCALLs are following the ADD.
nr_premapped is therere unchanged, since it does not change the consistency between M-EPT and S-EPT.
3. Unexpected zaps (such as kvm_zap_gfn_range()).
Those zaps are currently just paranoid ones. Not found in any existing paths yet. i.e.,
We want to detect any future code or any missed code piecies, which invokes kvm_zap_gfn_range() (or maybe zaps under read mmu_lock).
As those zaps do not necessarily hold slot_lock or filemap lock, they may ocurr after installing M-EPT and before installing S-EPT.
As a result, the BLOCK fails and tdx_is_sept_zap_err_due_to_premap() returns true.
Decreasing nr_premapped here to indicate the count of pages mapped in M-EPT but not in S-EPT decreases.
TDH.MEM.PAGE.ADD after this zap can still succeed. If this occurs, the page will be mapped in S-EPT only. As KVM also decreases nr_premapped after a successful TDH.MEM.PAGE.ADD, the nr_premapped will be <0 in the end. So, we will be able to detect those unexpected zaps.
When userspace invokes KVM_TDX_FINALIZE_VM,
=======
The nr_premapped must be 0 before tdx_td_finalize() succeeds.
The nr_premapped could be 0 if
(1) userspace invokes KVM_TDX_INIT_MEM_REGIONs as in a normal way.
(2) userspace never triggers any KVM_TDX_INIT_MEM_REGION.
(3) userspace triggers KVM_TDX_INIT_MEM_REGION but zaps all initial memory regions.
For (2)and(3), KVM_TDX_FINALIZE_VM can still succeed. So, TD can still run with uninitialized memory.
Quoted 8 lines
Quoted 4 lines
e.g., Before
KVM_TDX_FINALIZE_VM, if userspace performs a zap after the TDH.MEM.PAGE.ADD, the page will be removed from the S-EPT. The count ofnr_premappedwill not change after the successful TDH.MEM.RANGE.BLOCK and TDH.MEM.PAGE.REMOVE.Eww. It would be nice to close that hole, but I suppose it's futile, e.g. the underlying problem is unexpectedly removing pages from the initial, whether the VMM is doing stupid things before vs. after FINALIZE doesn't really matter.
Are you referring to the above "case 2 Expected zap paths"?
It's equal to that userspace never triggers any KVM_TDX_INIT_MEM_REGION. We can't force userspace must invoke KVM_TDX_INIT_MEM_REGION after all.
I don't think there's a hole from the guest point of view. See below.
Quoted 5 lines
Quoted 1 line
As a result, the TD will still run with uninitialized memory.
No? Because BLOCK+REMOVE means there are no valid S-EPT mappings. There's a "hole" that the guest might not expect, but that hole will trigger an EPT violation and only get "filled" if the guest explicitly accepts an AUG'd page.
If TD runs with unintialized memory,
- for the linux guest, it will cause TD to access unaccepted memory and get killed by KVM;
- for non-linux guest which configured with #VE, the guest will see #VE and be informed of that the page must be accepted before access. Though the guest should not be able to run without any initial code, but there's not any security problem.
Quoted 1 line
Side topic, why does KVM tolerate
tdh_mem_page_add()failure? IIUC, playing
We don't. It returns -EBUSY or -EIO immediately.
Quoted 3 lines
nice with
tdh_mem_page_add()failure necessitates both thetdx_is_sept_zap_err_due_to_premap()craziness and the check intdx_td_finalize()that all pending pages have been consumed.
tdx_is_sept_zap_err_due_to_premap() detects the error of BLOCK, which is caused by executing BLOCK before ADD.
Quoted 1 line
What reasonable use case is there for gracefully handling
tdh_mem_page_add()failure?
If tdh_mem_page_add() fails, the KVM_TDX_INIT_MEM_REGION just fails.
Quoted 3 lines
If there is a need to handle failure, I gotta imagine it's only for the
-EBUSYcase. And if it's only for-EBUSY, why can't that be handled by retrying intdx_vcpu_init_mem_region()? Iftdx_vcpu_init_mem_region()guarantees that all
I analyzed the contention status of tdh_mem_sept_add() at
https://lore.kernel.org/kvm/[email protected].
As the userspace is expected to execute KVM_TDX_INIT_MEM_REGION in only one vCPU, returning -EBUSY instead of retrying looks safer and easier.
Quoted 4 lines
pages mapped into the S-EPT are ADDed, then it can assert that there are no pending pages when it completes (even if it "fails"), and similarly
tdx_td_finalize()can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
tdx_td_finalize() now just returns -EINVAL in case of nr_premapped being !0. KVM_BUG_ON/WARN_ON should be also ok.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
arch/x86/kvm/vmx/tdx.c +10
-0+*/ +if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) { +if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) +return -EIO ; + +atomic64_inc (&kvm_tdx->nr_premapped); +return 0 ; + } + +return tdx_mem_page_aug (kvm, gfn, level, pfn); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,Signature · 1 line
2.51.0.268.g9569e192d0-goog
Sean Christopherson
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 13 lines · Yan Zhao
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Yan Zhao
Review static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1624
Quoted 8 lines · Sean Christopherson
(quoted diff) +7
-0@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, + /* + * If the TD isn't finalized/runnable, then userspace is initializing + * the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of + * pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD + * requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks + * the counter to ensure all mapped pages have been added to the image, + * to prevent running the TD with uninitialized memory.To prevent the mismatch between mirror EPT and the S-EPT?
Quoted 3 lines
Quoted 2 lines
No? Because KVM bumps the count when installing the S-EPT and decrements it on AUG, so I don't see how
nr_premappedguards against M-EPT vs. S-EPT issues?Hmm, I think there must be some misunderstanding.
Yeah, I forgot that AUG and ADD create the leaf S-EPT entries.
Quoted 33 lines
Before userspace invokes
KVM_TDX_FINALIZE_VM,
=======
1. the normal path (userspace invokesKVM_TDX_INIT_MEM_REGION).
(1) KVM holdsslot_lockand filemap lock.
(2) KVM invokeskvm_tdp_map_page()(orkvm_tdp_mmu_map_private_pfn()in patch 2).
KVM increasesnr_premappedintdx_sept_set_private_spte()to indicate that there's a page mapped in M-EPT, while it's not yet installed in S-EPT.
(3) KVM invokes TDH.MEM.PAGE.ADD and decreasesnr_premapped, indicating the page has been mapped in S-EPT too.As the name of
nr_premappedindicates, the count means a page is pre-mapped in the M-EPT, before its real mapping in the S-EPT.
If ADD fails in step (3),nr_premappedwill not be decreased.With mere the normal path,
nr_premappedshould return back to 0 after allKVM_TDX_INIT_MEM_REGIONs.2. Expected zap paths (e.g. If userspace does something strange, such as removing a slot after
KVM_TDX_INIT_MEM_REGION)
Those zap paths could be triggered by
1) userspace performs a page attribute conversion
2) userspace invokes gmem punch hole
3) userspace removes a slot
As all those paths either hold aslot_lockor a filemap lock, they can't contend withtdx_vcpu_init_mem_region()(tdx_vcpu_init_mem_regionholds bothslot_lockand internally filemap lock).
Consequently, those zaps must occur
a) beforekvm_tdp_map_page()or
b) after TDH.MEM.PAGE.ADD.
For a),tdx_sept_zap_private_spte()won't not be invoked as the page is not
Quoted 7 lines
mapped in M-EPT either;For b),
tdx_sept_zap_private_spte()should succeed, as the BLOCK and REMOVE
SEAMCALLs are following the ADD.
nr_premappedis therere unchanged, since it does not change the consistency between M-EPT and S-EPT.3. Unexpected zaps (such as
kvm_zap_gfn_range()).
Side topic related to kvm_zap_gfn_range(), the KVM_BUG_ON() in vt_refresh_apicv_exec_ctrl() is flawed. If kvm_recalculate_apic_map() fails to allocate an optimized map, KVM will mark APICv as inhibited, i.e. the associated WARN_ON_ONCE() is effectively user-triggerable.
Easiest thing would be to mark the vCPU as dead (though we obviously need "KVM: Never clear KVM_REQ_VM_DEAD from a vCPU's requests" for that to be robust).
arch/x86/kvm/vmx/main.c +2 -1
@@ -719,7 +719,8 @@ static void vt_set_apic_access_page_addr(struct kvm_vcpu *vcpu)static void vt_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) { -KVM_BUG_ON (!kvm_vcpu_apicv_active (vcpu), vcpu->kvm); +if (!kvm_vcpu_apicv_active (vcpu)) +kvm_make_request (KVM_REQ_VM_DEAD , vcpu);return ; }
Quoted 22 lines
Those zaps are currently just paranoid ones. Not found in any existing paths yet. i.e.,
We want to detect any future code or any missed code piecies, which invokeskvm_zap_gfn_range()(or maybe zaps under readmmu_lock).As those zaps do not necessarily hold
slot_lockor filemap lock, they may ocurr after installing M-EPT and before installing S-EPT.
As a result, the BLOCK fails andtdx_is_sept_zap_err_due_to_premap()returns true.
Decreasingnr_premappedhere to indicate the count of pages mapped in M-EPT but not in S-EPT decreases.TDH.MEM.PAGE.ADD after this zap can still succeed. If this occurs, the page will be mapped in S-EPT only. As KVM also decreases
nr_premappedafter a successful TDH.MEM.PAGE.ADD, thenr_premappedwill be <0 in the end. So, we will be able to detect those unexpected zaps.When userspace invokes
KVM_TDX_FINALIZE_VM,
=======
Thenr_premappedmust be 0 beforetdx_td_finalize()succeeds.
Quoted 7 lines
The
nr_premappedcould be 0 if
(1) userspace invokesKVM_TDX_INIT_MEM_REGIONsas in a normal way.
(2) userspace never triggers anyKVM_TDX_INIT_MEM_REGION.
(3) userspace triggersKVM_TDX_INIT_MEM_REGIONbut zaps all initial memory regions.For (2)and(3),
KVM_TDX_FINALIZE_VMcan still succeed.
Ya, we're in agreement on what can happen. I think all of the confusion was due to me forgetting that TDH.MEM.PAGE.ADD is what actually installes the leaf S-EPT entry.
Quoted 1 line
So, TD can still run with uninitialized memory.
No, the TD can never run with truly uninitialized memory. By "uninitialized", I mean memory that the guest can access and which has not been written to. Again, my confusion was due to thinking a page was already mapped into the guest, but awaiting TDH.MEM.PAGE.ADD to
Quoted 2 lines
Quoted 1 line
Side topic, why does KVM tolerate
tdh_mem_page_add()failure? IIUC, playingWe don't. It returns
-EBUSYor-EIOimmediately.
But that _is_ tolerating failure, in the sense that KVM doesn't prevent further actions on the VM. Tolerating failure is fine in general, but in this case it leaves the MMU is left in a half-baked state.
Quoted 6 lines
Quoted 3 lines
nice with
tdh_mem_page_add()failure necessitates both thetdx_is_sept_zap_err_due_to_premap()craziness and the check intdx_td_finalize()that all pending pages have been consumed.
tdx_is_sept_zap_err_due_to_premap()detects the error of BLOCK, which is caused by executing BLOCK before ADD.
We need to make this situation impossible.
Quoted 12 lines
Quoted 1 line
What reasonable use case is there for gracefully handling
tdh_mem_page_add()failure?If
tdh_mem_page_add()fails, theKVM_TDX_INIT_MEM_REGIONjust fails.Quoted 3 lines
If there is a need to handle failure, I gotta imagine it's only for the
-EBUSYcase. And if it's only for-EBUSY, why can't that be handled by retrying intdx_vcpu_init_mem_region()? Iftdx_vcpu_init_mem_region()guarantees that allI analyzed the contention status of
tdh_mem_sept_add()at
https://lore.kernel.org/kvm/[email protected].As the userspace is expected to execute
KVM_TDX_INIT_MEM_REGIONin only one vCPU, returning-EBUSYinstead of retrying looks safer and easier.
Quoted 6 lines
Quoted 4 lines
pages mapped into the S-EPT are ADDed, then it can assert that there are no pending pages when it completes (even if it "fails"), and similarly
tdx_td_finalize()can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
tdx_td_finalize()now just returns-EINVALin case ofnr_premappedbeing !0. KVM_BUG_ON/WARN_ON should be also ok.
Ok, so I vaguely recall that I may have pushed back on using a scratch field in "struct kvm_tdx" for temporary data (or maybe it was abusing vcpus[0] that I disliked?), but what we ended up with is far worse.
For all intents and purposes, nr_premapped _is_ a temporary scratch field, but with access rules that are all but impossible to understand, e.g. there's practically zero chance anyone could suss out complications with "Unexpected zaps", and the existence of that super subtle edge case necessitates using an atomic because KVM can't strictly guarantee that access to the field is mutually exclusive. And that also means it's inherently racy, e.g. if a zap happens while tdx_td_finalize() is checking nr_premapped, what happens?
The real killer is that deferring TDH.MEM.PAGE.ADD and TDH.MR_EXTEND until after the map completes and mmu_lock is dropped means that failure at that point leaves the TDP MMU in an inconsistent state, where the M-EPT has a present page but the S-EPT does not. Eww.
Note, in no way am I trying to blame anyone; quite the opposite, you've done an admirable job to get all of this landed. And I apologize if any of my past feedback led y'all down this road. I suspect my prefaulting idea really screwed things up; sorry :-(
Back to the code, unless I'm missing yet another complication, I think the obvious fix to all of this is to pass the source page and metadata flags via a scratch field in "struct kvm_tdx", and then do PAGE.ADD and MR.EXTEND in tdx_sept_set_private_spte(). Then there is no need to keep track of pending pages, because the M-EPT and S-EPT are always consistent. E.g. if PAGE.ADD fails with -EBUSY, then KVM will naturally revert the M-EPT entry from FROZEN to !PRESENT. It also allows KVM to KVM_BUG_ON() MR.EXTEND failure, because it should be impossible for the S-EPT entry to be modified between PAGE.ADD and MR.EXTEND.
Diff on top below for feedback on the idea. A proper series for this would simply replace several of the patches, e.g. asserting that slots_lock is held on tdx_is_sept_zap_err_due_to_premap() is wrong.
--- arch/x86/kvm/vmx/tdx.c | 157 +++++++++++++++++------------------------ arch/x86/kvm/vmx/tdx.h | 11 +-- 2 files changed, 70 insertions(+), 98 deletions(-)
arch/x86/kvm/vmx/tdx.c +64 -93
207 lines not on this page · raw message
arch/x86/kvm/vmx/tdx.h +6 -5
@@ -21,6 +21,8 @@ enum kvm_tdx_state { TD_STATE_RUNNABLE, }; +struct kvm_tdx_add_page ; +struct kvm_tdx {struct kvm kvm;
@@ -37,12 +39,11 @@ struct kvm_tdx {struct tdx_td td;/* -* The number of pages that KVM_TDX_INIT_MEM_REGION has mapped into the -* S-EPT, but not yet initialized via TDH.MEM.PAGE_ADD. Used to sanity -* check adding pages to the image, and to ensure that all pages have -* been initialized before finalizing the TD. +* Scratch structure used to pass the source page and metadata flags to +* tdx_mem_page_add. Protected by slots_lock, and non-NULL only when +* mapping a private pfn via tdx_gmem_post_populate(). */ -unsigned long nr_pending_tdh_mem_page_adds; +struct kvm_tdx_page_add *page_add;/* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do
base-commit: 7c7a3893b102bdeb4826f7140280b7b16081b385 --
Edgecombe, Rick P
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 61 lines · Sean Christopherson
Quoted 13 lines · Yan Zhao
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Yan Zhao
Review static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1624
Quoted 8 lines · Sean Christopherson
(quoted diff) +7
-0@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, + /* + * If the TD isn't finalized/runnable, then userspace is initializing + * the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of + * pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD + * requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks + * the counter to ensure all mapped pages have been added to the image, + * to prevent running the TD with uninitialized memory.To prevent the mismatch between mirror EPT and the S-EPT?
Quoted 3 lines
Quoted 2 lines
No? Because KVM bumps the count when installing the S-EPT and decrements it on AUG, so I don't see how
nr_premappedguards against M-EPT vs. S-EPT issues?Hmm, I think there must be some misunderstanding.
Yeah, I forgot that AUG and ADD create the leaf S-EPT entries.
Quoted 33 lines
Before userspace invokes
KVM_TDX_FINALIZE_VM,
=======
1. the normal path (userspace invokesKVM_TDX_INIT_MEM_REGION).
(1) KVM holdsslot_lockand filemap lock.
(2) KVM invokeskvm_tdp_map_page()(orkvm_tdp_mmu_map_private_pfn()in patch 2).
KVM increasesnr_premappedintdx_sept_set_private_spte()to indicate that there's a page mapped in M-EPT, while it's not yet installed in S-EPT.
(3) KVM invokes TDH.MEM.PAGE.ADD and decreasesnr_premapped, indicating the page has been mapped in S-EPT too.As the name of
nr_premappedindicates, the count means a page is pre-mapped in the M-EPT, before its real mapping in the S-EPT.
If ADD fails in step (3),nr_premappedwill not be decreased.With mere the normal path,
nr_premappedshould return back to 0 after allKVM_TDX_INIT_MEM_REGIONs.2. Expected zap paths (e.g. If userspace does something strange, such as removing a slot after
KVM_TDX_INIT_MEM_REGION)
Those zap paths could be triggered by
1) userspace performs a page attribute conversion
2) userspace invokes gmem punch hole
3) userspace removes a slot
As all those paths either hold aslot_lockor a filemap lock, they can't contend withtdx_vcpu_init_mem_region()(tdx_vcpu_init_mem_regionholds bothslot_lockand internally filemap lock).
Consequently, those zaps must occur
a) beforekvm_tdp_map_page()or
b) after TDH.MEM.PAGE.ADD.
For a),tdx_sept_zap_private_spte()won't not be invoked as the page is notQuoted 7 lines
mapped in M-EPT either;For b),
tdx_sept_zap_private_spte()should succeed, as the BLOCK and REMOVE
SEAMCALLs are following the ADD.
nr_premappedis therere unchanged, since it does not change the consistency between M-EPT and S-EPT.3. Unexpected zaps (such as
kvm_zap_gfn_range()).
Quoted 10 lines
Side topic related to
kvm_zap_gfn_range(), theKVM_BUG_ON()invt_refresh_apicv_exec_ctrl()is flawed. Ifkvm_recalculate_apic_map()fails to allocate an optimized map, KVM will mark APICv as inhibited, i.e. the associatedWARN_ON_ONCE()is effectively user-triggerable.Easiest thing would be to mark the vCPU as dead (though we obviously need "KVM: Never clear
KVM_REQ_VM_DEADfrom a vCPU's requests" for that to be robust).
I'm going need to look up the related apic discussions from the base series and circle back.
[snip]
Quoted 6 lines
Quoted 2 lines
tdx_td_finalize()now just returns-EINVALin case ofnr_premappedbeing !0. KVM_BUG_ON/WARN_ON should be also ok.Ok, so I vaguely recall that I may have pushed back on using a scratch field in "struct
kvm_tdx" for temporary data (or maybe it was abusing vcpus[0] that I disliked?), but what we ended up with is far worse.
I think it was also that the tdh_mr_extend() loop was too heavyweight for the fault path. But that was before we got to the kick+lock stuff.
Quoted 14 lines
For all intents and purposes,
nr_premapped_is_ a temporary scratch field, but with access rules that are all but impossible to understand, e.g. there's practically zero chance anyone could suss out complications with "Unexpected zaps", and the existence of that super subtle edge case necessitates using an atomic because KVM can't strictly guarantee that access to the field is mutually exclusive. And that also means it's inherently racy, e.g. if a zap happens whiletdx_td_finalize()is checkingnr_premapped, what happens?The real killer is that deferring TDH.MEM.PAGE.ADD and TDH.MR_EXTEND until after the map completes and
mmu_lockis dropped means that failure at that point leaves the TDP MMU in an inconsistent state, where the M-EPT has a present page but the S-EPT does not. Eww.
Quoted 4 lines
Note, in no way am I trying to blame anyone; quite the opposite, you've done an admirable job to get all of this landed. And I apologize if any of my past feedback led y'all down this road. I suspect my prefaulting idea really screwed things up; sorry :-(
It's unfortunate we didn't have the gmem mmap() support then. Otherwise we could have just encrypted it in place.
Otherwise, I'm really glad to see these cleanups/scrutiny. I kind of got the impression that you wanted to see less TDX churn for a bit. The fact is, the TDX base support still needs more work like this.
Quoted 10 lines
Back to the code, unless I'm missing yet another complication, I think the obvious fix to all of this is to pass the source page and metadata flags via a scratch field in "struct
kvm_tdx", and then do PAGE.ADD and MR.EXTEND intdx_sept_set_private_spte(). Then there is no need to keep track of pending pages, because the M-EPT and S-EPT are always consistent. E.g. if PAGE.ADD fails with-EBUSY, then KVM will naturally revert the M-EPT entry from FROZEN to !PRESENT. It also allows KVM toKVM_BUG_ON()MR.EXTEND failure, because it should be impossible for the S-EPT entry to be modified between PAGE.ADD and MR.EXTEND.
Quoted 3 lines
Diff on top below for feedback on the idea. A proper series for this would simply replace several of the patches, e.g. asserting that
slots_lockis held ontdx_is_sept_zap_err_due_to_premap()is wrong.
This works. The "stuffing data on the vCPU" part is a little ugly as it was before, but the other solutions were worse. Especially nr_premap was problematic for a number of reasons that keeps growing.
So, big Acked-by: Rick Edgecombe <[email protected]>
Quoted 256 lines
--- arch/x86/kvm/vmx/tdx.c | 157 +++++++++++++++++------------------------ arch/x86/kvm/vmx/tdx.h | 11 +-- 2 files changed, 70 insertions(+), 98 deletions(-)arch/x86/kvm/vmx/tdx.c +64
-93@@ -1586,6 +1586,56 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } + +struct kvm_tdx_page_add { +struct page *src; +unsigned long flags; +}; + +static int tdx_mem_page_add (struct kvm *kvm,gfn_t gfn,enum pg_level level, +kvm_pfn_t pfn) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err, entry, level_state; +gpa_t gpa =gfn_to_gpa (gfn); +int i; + +lockdep_assert_held (&kvm->slots_lock); + +if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm) || +KVM_BUG_ON (!kvm_tdx->page_add, kvm)) +return -EIO ; + + err =tdh_mem_page_add (&kvm_tdx->td, gpa,pfn_to_page (pfn), + kvm_tdx->page_add->src, &entry, &level_state); +if (unlikely (tdx_operand_busy (err))) +return -EBUSY ; + +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error_2 (TDH_MEM_PAGE_ADD , err, entry, level_state); +return -EIO ; + } + +if (!(kvm_tdx->page_add->flags &KVM_TDX_MEASURE_MEMORY_REGION )) +return 0 ; + +/* +* Extend the measurement while holding mmu_lock to ensure MR.EXTEND +* can't fail, e.g. due to the S-EPT entry being zapped after PAGE.ADD. +* Note! If extended the measurement fails, bug the VM, but do NOT +* return an error, as mapping the page in the S-EPT succeeded and so +* needs to be tracked in KVM's mirror page tables. +*/ +for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) { + err =tdh_mr_extend (&kvm_tdx->td, gpa + i, &entry, &level_state); +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error_2 (TDH_MR_EXTEND , err, entry, level_state); +break ; + } + } +return 0 ; +} +static int tdx_mem_page_aug (struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn) {@@ -1627,21 +1677,11 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,/* * If the TD isn't finalized/runnable, then userspace is initializing -* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of -* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD -* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks -* the counter to ensure all mapped pages have been added to the image, -* to prevent running the TD with uninitialized memory. +* the VM image via KVM_TDX_INIT_MEM_REGION. Add the page to the TD, +* and optionally extend the measurement with the page contents. */ -if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) { -lockdep_assert_held (&kvm->slots_lock); - -if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EIO ; - - kvm_tdx->nr_pending_tdh_mem_page_adds++; -return 0 ; - } +if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) +return tdx_mem_page_add (kvm, gfn, level, pfn);return tdx_mem_page_aug (kvm, gfn, level, pfn); }@@ -1716,39 +1756,6 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,return 0 ; } -/* -* Check if the error returned from a SEPT zap SEAMCALL is due to that a page is -* mapped by KVM_TDX_INIT_MEM_REGION without tdh_mem_page_add() being called -* successfully. -* -* Since tdh_mem_sept_add() must have been invoked successfully before a -* non-leaf entry present in the mirrored page table, the SEPT ZAP related -* SEAMCALLs should not encounter err TDX_EPT_WALK_FAILED. They should instead -* find TDX_EPT_ENTRY_STATE_INCORRECT due to an empty leaf entry found in the -* SEPT. -* -* Further check if the returned entry from SEPT walking is with RWX permissions -* to filter out anything unexpected. -* -* Note: @level is pg_level, not the tdx_level. The tdx_level extracted from -* level_state returned from a SEAMCALL error is the same as that passed into -* the SEAMCALL. -*/ -static int tdx_is_sept_zap_err_due_to_premap (struct kvm_tdx *kvm_tdx,u64 err, -u64 entry,int level) -{ -if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE ) -return false; - -if (err != (TDX_EPT_ENTRY_STATE_INCORRECT |TDX_OPERAND_ID_RCX )) -return false; - -if ((is_last_spte (entry, level) && (entry &VMX_EPT_RWX_MASK ))) -return false; - -return true; -} -static int tdx_sept_zap_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {@@ -1768,15 +1775,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, err =tdh_mem_range_block (&kvm_tdx->td, gpa, tdx_level, &entry, &level_state);tdx_no_vcpus_enter_stop (kvm); } -if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) { -lockdep_assert_held (&kvm->slots_lock); - -if (KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm)) -return -EIO ; - -return 0 ; - } -if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_RANGE_BLOCK , err, entry, level_state);return -EIO ;@@ -2842,12 +2840,6 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd)if (!is_hkid_assigned (kvm_tdx) || kvm_tdx->state ==TD_STATE_RUNNABLE )return -EINVAL ; -/* -* Pages are pending for KVM_TDX_INIT_MEM_REGION to issue -* TDH.MEM.PAGE.ADD(). -*/ -if (kvm_tdx->nr_pending_tdh_mem_page_adds) -return -EINVAL ; cmd->hw_error =tdh_mr_finalize (&kvm_tdx->td);if (tdx_operand_busy (cmd->hw_error))@@ -3131,50 +3123,29 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, {struct tdx_gmem_post_populate_arg *arg = _arg;struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); -u64 err, entry, level_state; -gpa_t gpa =gfn_to_gpa (gfn); -struct page *src_page; -int ret, i; +struct kvm_tdx_page_add page_add = { + .flags = arg->flags, + }; +int ret; -lockdep_assert_held (&kvm->slots_lock); +if (KVM_BUG_ON (kvm_tdx->page_add, kvm)) +return -EIO ;/* * Get the source page if it has been faulted in. Return failure if the * source page has been swapped out or unmapped in primary memory. */ - ret =get_user_pages_fast ((unsigned long )src,1 ,0 , &src_page); + ret =get_user_pages_fast ((unsigned long )src,1 ,0 , &page_add.src);if (ret <0 )return ret;if (ret !=1 )return -ENOMEM ; + kvm_tdx->page_add = &page_add; ret =kvm_tdp_mmu_map_private_pfn (arg->vcpu, gfn, pfn); -if (ret <0 ) - goto out; + kvm_tdx->page_add =NULL ; - ret =0 ; - err =tdh_mem_page_add (&kvm_tdx->td, gpa,pfn_to_page (pfn), - src_page, &entry, &level_state); -if (err) { - ret =unlikely (tdx_operand_busy (err)) ? -EBUSY : -EIO ; - goto out; - } - -KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm); - -if (arg->flags &KVM_TDX_MEASURE_MEMORY_REGION ) { -for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) { - err =tdh_mr_extend (&kvm_tdx->td, gpa + i, &entry, - &level_state); -if (err) { - ret = -EIO ; -break ; - } - } - } - -out: -put_page (src_page); +put_page (page_add.src);return ret; }arch/x86/kvm/vmx/tdx.h +6
-5@@ -21,6 +21,8 @@ enum kvm_tdx_state { TD_STATE_RUNNABLE, }; +struct kvm_tdx_add_page ; +struct kvm_tdx {struct kvm kvm;@@ -37,12 +39,11 @@ struct kvm_tdx {struct tdx_td td;/* -* The number of pages that KVM_TDX_INIT_MEM_REGION has mapped into the -* S-EPT, but not yet initialized via TDH.MEM.PAGE_ADD. Used to sanity -* check adding pages to the image, and to ensure that all pages have -* been initialized before finalizing the TD. +* Scratch structure used to pass the source page and metadata flags to +* tdx_mem_page_add. Protected by slots_lock, and non-NULL only when +* mapping a private pfn via tdx_gmem_post_populate(). */ -unsigned long nr_pending_tdh_mem_page_adds; +struct kvm_tdx_page_add *page_add;/* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do base-commit:
7c7a3893b102bdeb4826f7140280b7b16081b385--
Sean Christopherson
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 10 lines · Rick P Edgecombe
Quoted 6 lines · Sean Christopherson
Quoted 2 lines
tdx_td_finalize()now just returns-EINVALin case ofnr_premappedbeing !0. KVM_BUG_ON/WARN_ON should be also ok.Ok, so I vaguely recall that I may have pushed back on using a scratch field in "struct
kvm_tdx" for temporary data (or maybe it was abusing vcpus[0] that I disliked?), but what we ended up with is far worse.I think it was also that the
tdh_mr_extend()loop was too heavyweight for the fault path. But that was before we got to the kick+lock stuff.
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
If tdh_mr_extend() is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are no cond_resched() calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicit cond_resched(), there's no practical difference between extending the measurement under mmu_lock versus outside of mmu_lock.
_If_ we need/want to do tdh_mr_extend() outside of mmu_lock, we can and should still do tdh_mem_page_add() under mmu_lock.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 2 lines · Sean Christopherson
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
Just was my recollection of the discussion. I found it:
https://lore.kernel.org/lkml/[email protected]/
Quoted 9 lines
If
tdh_mr_extend()is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are nocond_resched()calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicitcond_resched(), there's no practical difference between extending the measurement undermmu_lockversus outside ofmmu_lock._If_ we need/want to do
tdh_mr_extend()outside ofmmu_lock, we can and should still dotdh_mem_page_add()undermmu_lock.
I just did a quick test and we should be on the order of <1 ms per page for the full loop. I can try to get some more formal test data if it matters. But that doesn't sound too horrible?
tdh_mr_extend() outside MMU lock is tempting because it doesn't *need* to be inside it. But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error in tdx_mem_page_add() due to code in another file).
I wonder if Yan can give an analysis of any zapping races if we do that.
Sean Christopherson
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 6 lines · Rick P Edgecombe
Quoted 2 lines · Sean Christopherson
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
Just was my recollection of the discussion. I found it:
https://lore.kernel.org/lkml/[email protected]/
Ugh, another case where an honest question gets interpreted as "do it this way". :-(
Quoted 6 lines
Quoted 6 lines
If
tdh_mr_extend()is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are nocond_resched()calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicitcond_resched(), there's no practical difference between extending the measurement undermmu_lockversus outside ofmmu_lock.
Quoted 6 lines
Quoted 2 lines
_If_ we need/want to do
tdh_mr_extend()outside ofmmu_lock, we can and should still dotdh_mem_page_add()undermmu_lock.I just did a quick test and we should be on the order of <1 ms per page for the full loop. I can try to get some more formal test data if it matters. But that doesn't sound too horrible?
1ms is totally reasonable. I wouldn't bother with any more testing.
Quoted 2 lines
tdh_mr_extend()outside MMU lock is tempting because it doesn't *need* to be inside it.
Agreed, and it would eliminate the need for a "flags" argument. But keeping it in the mmu_lock critical section means KVM can WARN on failures. If it's moved out, then zapping S-EPT entries could induce failure, and I don't think it's worth going through the effort to ensure it's impossible to trigger S-EPT removal.
Note, temoving S-EPT entries during initialization of the image isn't something I want to official support, rather it's an endless stream of whack-a-mole due to obsurce edge cases
Hmm, actually, maybe I take that back. slots_lock prevents memslot updates, filemap_invalidate_lock() prevents guest_memfd updates, and mmu_notifier events shouldn't ever hit S-EPT. I was worried about kvm_zap_gfn_range(), but the call from sev.c is obviously mutually exclusive, TDX disallows KVM_X86_QUIRK_IGNORE_GUEST_PAT so same goes for kvm_noncoherent_dma_assignment_start_or_stop, and while I'm 99% certain there's a way to trip __kvm_set_or_clear_apicv_inhibit(), the APIC page has its own non-guest_memfd memslot and so can't be used for the initial image, which means that too is mutually exclusive.
So yeah, let's give it a shot. Worst case scenario we're wrong and TDH_MR_EXTEND errors can be triggered by userspace.
Quoted 5 lines
But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error in
tdx_mem_page_add()due to code in another file).I wonder if Yan can give an analysis of any zapping races if we do that.
As above, I think we're good?
Edgecombe, Rick P
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 9 lines · Sean Christopherson
Agreed, and it would eliminate the need for a "flags" argument. But keeping it in the
mmu_lockcritical section means KVM can WARN on failures. If it's moved out, then zapping S-EPT entries could induce failure, and I don't think it's worth going through the effort to ensure it's impossible to trigger S-EPT removal.Note, temoving S-EPT entries during initialization of the image isn't something I want to official support, rather it's an endless stream of whack-a-mole due to obsurce edge cases
Quoted 6 lines
Hmm, actually, maybe I take that back.
slots_lockprevents memslot updates,filemap_invalidate_lock()preventsguest_memfdupdates, andmmu_notifierevents shouldn't ever hit S-EPT. I was worried aboutkvm_zap_gfn_range(), but the call fromsev.cis obviously mutually exclusive, TDX disallowsKVM_X86_QUIRK_IGNORE_GUEST_PATso same goes forkvm_noncoherent_dma_assignment_start_or_stop, and
Yea, in the other thread Yan was suggesting the same thing from the KVM side:
https://lore.kernel.org/all/aK%[email protected]/
But was concerned about "Unexpected zaps" (kvm_zap_gfn_range()). I think maybe we could think about KVM_BUG_ON() in the case of mirror EPT to cover it from another angle. IIRC we discussed this at some point.
I was wondering about TDH.MR.EXTEND error conditions. Coming back now, I'm not sure what I was thinking.
Quoted 3 lines
while I'm 99% certain there's a way to trip
__kvm_set_or_clear_apicv_inhibit(), the APIC page has its own non-guest_memfd memslot and so can't be used for the initial image, which means that too is mutually exclusive.
Hmm, well maybe KVM_BUG_ON() for kvm_zap_gfn_range() only if this gets addressed.
Quoted 11 lines
So yeah, let's give it a shot. Worst case scenario we're wrong and
TDH_MR_EXTENDerrors can be triggered by userspace.Quoted 5 lines
But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error in
tdx_mem_page_add()due to code in another file).I wonder if Yan can give an analysis of any zapping races if we do that.
As above, I think we're good?
Works for me.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Sean ChristophersonQuoted 49 lines · Sean Christopherson
Quoted 6 lines · Rick P Edgecombe
Quoted 2 lines · Sean Christopherson
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
Just was my recollection of the discussion. I found it:
https://lore.kernel.org/lkml/[email protected]/Ugh, another case where an honest question gets interpreted as "do it this way". :-(
Quoted 6 lines
Quoted 6 lines
If
tdh_mr_extend()is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are nocond_resched()calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicitcond_resched(), there's no practical difference between extending the measurement undermmu_lockversus outside ofmmu_lock.Quoted 6 lines
Quoted 2 lines
_If_ we need/want to do
tdh_mr_extend()outside ofmmu_lock, we can and should still dotdh_mem_page_add()undermmu_lock.I just did a quick test and we should be on the order of <1 ms per page for the full loop. I can try to get some more formal test data if it matters. But that doesn't sound too horrible?
1ms is totally reasonable. I wouldn't bother with any more testing.
Quoted 2 lines
tdh_mr_extend()outside MMU lock is tempting because it doesn't *need* to be inside it.Agreed, and it would eliminate the need for a "flags" argument. But keeping it in the
mmu_lockcritical section means KVM can WARN on failures. If it's moved out, then zapping S-EPT entries could induce failure, and I don't think it's worth going through the effort to ensure it's impossible to trigger S-EPT removal.Note, temoving S-EPT entries during initialization of the image isn't something I want to official support, rather it's an endless stream of whack-a-mole due to obsurce edge cases
Hmm, actually, maybe I take that back.
slots_lockprevents memslot updates,filemap_invalidate_lock()preventsguest_memfdupdates, andmmu_notifierevents shouldn't ever hit S-EPT. I was worried aboutkvm_zap_gfn_range(), but the call fromsev.cis obviously mutually exclusive, TDX disallowsKVM_X86_QUIRK_IGNORE_GUEST_PATso same goes forkvm_noncoherent_dma_assignment_start_or_stop, and while I'm 99% certain there's a way to trip__kvm_set_or_clear_apicv_inhibit(), the APIC page has its own non-guest_memfd memslot and so can't be used for the initial image, which means that too is mutually exclusive.So yeah, let's give it a shot. Worst case scenario we're wrong and
TDH_MR_EXTENDerrors can be triggered by userspace.
Quoted 7 lines
Quoted 5 lines
But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error in
tdx_mem_page_add()due to code in another file).I wonder if Yan can give an analysis of any zapping races if we do that.
As above, I think we're good?
I think so.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Edgecombe, Rick PQuoted 17 lines · Rick P
Quoted 2 lines · Sean Christopherson
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
Just was my recollection of the discussion. I found it:
https://lore.kernel.org/lkml/[email protected]/Quoted 9 lines
If
tdh_mr_extend()is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are nocond_resched()calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicitcond_resched(), there's no practical difference between extending the measurement undermmu_lockversus outside ofmmu_lock._If_ we need/want to do
tdh_mr_extend()outside ofmmu_lock, we can and should still dotdh_mem_page_add()undermmu_lock.
Quoted 6 lines
I just did a quick test and we should be on the order of <1 ms per page for the full loop. I can try to get some more formal test data if it matters. But that doesn't sound too horrible?
tdh_mr_extend()outside MMU lock is tempting because it doesn't *need* to be inside it.
I'm probably not following this conversation, so stupid question: It doesn't need to be in the lock because user space should not be setting up memory and extending the measurement in an asynchronous way. Is that correct?
Quoted 5 lines
But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error in
tdx_mem_page_add()due to code in another file).I wonder if Yan can give an analysis of any zapping races if we do that.
When you say analysis, you mean detecting user space did something wrong and failing gracefully? Is that correct?
Ira
Sean Christopherson
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 22 lines · Ira Weiny
Quoted 21 lines · Rick P
Quoted 2 lines · Sean Christopherson
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
Just was my recollection of the discussion. I found it:
https://lore.kernel.org/lkml/[email protected]/Quoted 9 lines
If
tdh_mr_extend()is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are nocond_resched()calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicitcond_resched(), there's no practical difference between extending the measurement undermmu_lockversus outside ofmmu_lock._If_ we need/want to do
tdh_mr_extend()outside ofmmu_lock, we can and should still dotdh_mem_page_add()undermmu_lock.I just did a quick test and we should be on the order of <1 ms per page for the full loop. I can try to get some more formal test data if it matters. But that doesn't sound too horrible?
Quoted 7 lines
Quoted 2 lines
tdh_mr_extend()outside MMU lock is tempting because it doesn't *need* to be inside it.I'm probably not following this conversation, so stupid question: It doesn't need to be in the lock because user space should not be setting up memory and extending the measurement in an asynchronous way. Is that correct?
No, from userspace's perspective ADD+MEASURE is fully serialized. ADD "needs" to be under mmu_lock to guarantee consistency between the mirror EPT and the "real" S-EPT entries. E.g. if ADD is done after the fact, then KVM can end up with a PRESENT M-EPT entry but a corresponding S-EPT entry that is !PRESENT. That causes a pile of problems because it breaks KVM's fundamental assumption that M-EPT and S-EPT entries updated in lock-step.
TDH_MR_EXTEND doesn't have the same same consistency issue. If it fails, the only thing that's left in a bad state is the measurement. That's obviously not ideal either, but we can handle that by forcefully terminating the VM, without opening up KVM to edge cases that would otherwise be impossible.
Quoted 8 lines
Quoted 5 lines
But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error in
tdx_mem_page_add()due to code in another file).I wonder if Yan can give an analysis of any zapping races if we do that.
When you say analysis, you mean detecting user space did something wrong and failing gracefully? Is that correct?
More specifically, whether or not KVM can WARN without the WARN being user triggerable. Kernel policy is that WARNs must not be triggerable absent kernel, hardware, or firmware bugs. What we're trying to figure out is if there's a flow that can be triggered by userspace (misbehving or not) that would trip a WARN even if KVM is operating as expected. I'm pretty sure the answer is "no".
Oh, and WARNing here is desirable, because it improves the chances of detecting a fatal-to-the-VM bug, e.g. in KVM and/or in the TDX-Module.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 31 lines · Sean Christopherson
Quoted 22 lines · Ira Weiny
Quoted 21 lines · Rick P
Quoted 2 lines · Sean Christopherson
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
Just was my recollection of the discussion. I found it:
https://lore.kernel.org/lkml/[email protected]/Quoted 9 lines
If
tdh_mr_extend()is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are nocond_resched()calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicitcond_resched(), there's no practical difference between extending the measurement undermmu_lockversus outside ofmmu_lock._If_ we need/want to do
tdh_mr_extend()outside ofmmu_lock, we can and should still dotdh_mem_page_add()undermmu_lock.I just did a quick test and we should be on the order of <1 ms per page for the full loop. I can try to get some more formal test data if it matters. But that doesn't sound too horrible?
Quoted 7 lines
Quoted 2 lines
tdh_mr_extend()outside MMU lock is tempting because it doesn't *need* to be inside it.I'm probably not following this conversation, so stupid question: It doesn't need to be in the lock because user space should not be setting up memory and extending the measurement in an asynchronous way. Is that correct?
Quoted 6 lines
No, from userspace's perspective ADD+MEASURE is fully serialized. ADD "needs" to be under
mmu_lockto guarantee consistency between the mirror EPT and the "real" S-EPT entries. E.g. if ADD is done after the fact, then KVM can end up with a PRESENT M-EPT entry but a corresponding S-EPT entry that is !PRESENT. That causes a pile of problems because it breaks KVM's fundamental assumption that M-EPT and S-EPT entries updated in lock-step.
Ok yes, I think I worded my query incorrectly but this makes things clear.
Thanks!
Quoted 15 lines
TDH_MR_EXTENDdoesn't have the same same consistency issue. If it fails, the only thing that's left in a bad state is the measurement. That's obviously not ideal either, but we can handle that by forcefully terminating the VM, without opening up KVM to edge cases that would otherwise be impossible.Quoted 8 lines
Quoted 5 lines
But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error in
tdx_mem_page_add()due to code in another file).I wonder if Yan can give an analysis of any zapping races if we do that.
When you say analysis, you mean detecting user space did something wrong and failing gracefully? Is that correct?
Quoted 8 lines
More specifically, whether or not KVM can WARN without the WARN being user triggerable. Kernel policy is that WARNs must not be triggerable absent kernel, hardware, or firmware bugs. What we're trying to figure out is if there's a flow that can be triggered by userspace (misbehving or not) that would trip a WARN even if KVM is operating as expected. I'm pretty sure the answer is "no".
Oh, and WARNing here is desirable, because it improves the chances of detecting a fatal-to-the-VM bug, e.g. in KVM and/or in the TDX-Module.
OK... In other areas of the kernel if the user misbehaves it is reasonable to fail an operation. I would think that being fatal to the VM would be fine if QEMU did not properly synchronize ADD, measurement, and finalize, for example. Am I wrong in that assumption?
Ira
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Edgecombe, Rick PQuoted 21 lines · Rick P
Quoted 2 lines · Sean Christopherson
Me confused. This is pre-boot, not the normal fault path, i.e. blocking other operations is not a concern.
Just was my recollection of the discussion. I found it:
https://lore.kernel.org/lkml/[email protected]/Quoted 9 lines
If
tdh_mr_extend()is too heavy for a non-preemptible section, then the current code is also broken in the sense that there are nocond_resched()calls. The vast majority of TDX hosts will be using non-preemptible kernels, so without an explicitcond_resched(), there's no practical difference between extending the measurement undermmu_lockversus outside ofmmu_lock._If_ we need/want to do
tdh_mr_extend()outside ofmmu_lock, we can and should still dotdh_mem_page_add()undermmu_lock.I just did a quick test and we should be on the order of <1 ms per page for the full loop. I can try to get some more formal test data if it matters. But that doesn't sound too horrible?
Quoted 6 lines
tdh_mr_extend()outside MMU lock is tempting because it doesn't *need* to be inside it. But maybe a better reason is that we could better handle errors outside the fault. (i.e. no 5 line comment about why not to return an error intdx_mem_page_add()due to code in another file).I wonder if Yan can give an analysis of any zapping races if we do that.
I actually proposed to have write mmu_lock around tdh_mem_page_add() and tdh_mr_extend(), as in
https://lore.kernel.org/kvm/[email protected].
I don't see any reason why tdh_mr_extend() can't be done inside mmu_lock in the pre-boot stage.
But the previous conclusion was that with slots_lock and filemap invalidation lock, it's ok to invoke tdh_mem_page_add() and tdh_mr_extend() without any mmu_lock. The nr_premapped can also detect the unexpected zap.
Sean Christopherson
maintainerRe: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Edgecombe, Rick PQuoted 2 lines · Rick P Edgecombe
It's unfortunate we didn't have the gmem
mmap()support then. Otherwise we could have just encrypted it in place.
Yeah, hindsight is definitely 20/20 on that front. Though I suspect that we'd never have landed anything if we tried to go straight to support mmap().
Quoted 2 lines
Otherwise, I'm really glad to see these cleanups/scrutiny. I kind of got the impression that you wanted to see less TDX churn for a bit.
Heh, believe me, I'm not exactly ecstatic to dive into this. But, I don't want to just ignore it, because some of these quirks/warts are already getting in the way of new development, and if I/we delay such clean ups, the pain is only going to get worse (and the total cost will be much higher).
Fatigue is a bit of a problem for me, but the biggest issue really is just lack of cycles (the quick feedback and testing y'all are providing helps tremendously on that front). And lack of cycles should be mitigated to some extent as I (re)familiarize myself with the code; I recognized most of the concepts, but there are definitely a few places where I'm completely lost, and that makes reviewing things like dynamic PAMT and hugepage support excrutiatingly slow.
Quoted 1 line
The fact is, the TDX base support still needs more work like this.
IMO, the most important things to address are cases where the design choices we made turned out to be suboptimal, i.e. where the behavior itself is problematic. Code cleanups are definitely welcome too, but my priority is polishing the core design.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Edgecombe, Rick PQuoted 2 lines · Rick P
Quoted 1 line · Sean Christopherson
On Thu, Aug 28, 2025, Yan Zhao wrote:
[...]
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
3. Unexpected zaps (such as
kvm_zap_gfn_range()).Side topic related to
kvm_zap_gfn_range(), theKVM_BUG_ON()invt_refresh_apicv_exec_ctrl()is flawed. Ifkvm_recalculate_apic_map()fails to allocate an optimized map, KVM will mark APICv as inhibited, i.e. the associatedWARN_ON_ONCE()is effectively user-triggerable.
Quoted 7 lines
Quoted 5 lines
Easiest thing would be to mark the vCPU as dead (though we obviously need "KVM: Never clear
KVM_REQ_VM_DEADfrom a vCPU's requests" for that to be robust).I'm going need to look up the related apic discussions from the base series and circle back.
There was an analysis about the inhibit reasons for TDX.
https://lore.kernel.org/lkml/[email protected]/
As Sean mentioned, if kvm_recalculate_apic_map() fails to allocate the memory for optimized map, it will trigger the KVM_BUG_ON() in vt_refresh_apicv_exec_ctrl(). And kvzalloc() failure should not be treated as KVM bug.
As talking about user-triggerable, the kvzalloc() failure path could be triggered by KVM_CREATE_VCPU and KVM_TDX_INIT_VCPU for TD. After KVM_TDX_INIT_VCPU, the mapping is not allowed to be changed.
Sean's suggested code change looks good to me.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Sean ChristophersonQuoted 131 lines · Sean Christopherson
Quoted 13 lines · Yan Zhao
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Yan Zhao
Review static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1624
Quoted 8 lines · Sean Christopherson
(quoted diff) +7
-0@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, + /* + * If the TD isn't finalized/runnable, then userspace is initializing + * the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of + * pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD + * requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks + * the counter to ensure all mapped pages have been added to the image, + * to prevent running the TD with uninitialized memory.To prevent the mismatch between mirror EPT and the S-EPT?
Quoted 3 lines
Quoted 2 lines
No? Because KVM bumps the count when installing the S-EPT and decrements it on AUG, so I don't see how
nr_premappedguards against M-EPT vs. S-EPT issues?Hmm, I think there must be some misunderstanding.
Yeah, I forgot that AUG and ADD create the leaf S-EPT entries.
Quoted 33 lines
Before userspace invokes
KVM_TDX_FINALIZE_VM,
=======
1. the normal path (userspace invokesKVM_TDX_INIT_MEM_REGION).
(1) KVM holdsslot_lockand filemap lock.
(2) KVM invokeskvm_tdp_map_page()(orkvm_tdp_mmu_map_private_pfn()in patch 2).
KVM increasesnr_premappedintdx_sept_set_private_spte()to indicate that there's a page mapped in M-EPT, while it's not yet installed in S-EPT.
(3) KVM invokes TDH.MEM.PAGE.ADD and decreasesnr_premapped, indicating the page has been mapped in S-EPT too.As the name of
nr_premappedindicates, the count means a page is pre-mapped in the M-EPT, before its real mapping in the S-EPT.
If ADD fails in step (3),nr_premappedwill not be decreased.With mere the normal path,
nr_premappedshould return back to 0 after allKVM_TDX_INIT_MEM_REGIONs.2. Expected zap paths (e.g. If userspace does something strange, such as removing a slot after
KVM_TDX_INIT_MEM_REGION)
Those zap paths could be triggered by
1) userspace performs a page attribute conversion
2) userspace invokes gmem punch hole
3) userspace removes a slot
As all those paths either hold aslot_lockor a filemap lock, they can't contend withtdx_vcpu_init_mem_region()(tdx_vcpu_init_mem_regionholds bothslot_lockand internally filemap lock).
Consequently, those zaps must occur
a) beforekvm_tdp_map_page()or
b) after TDH.MEM.PAGE.ADD.
For a),tdx_sept_zap_private_spte()won't not be invoked as the page is notQuoted 7 lines
mapped in M-EPT either;For b),
tdx_sept_zap_private_spte()should succeed, as the BLOCK and REMOVE
SEAMCALLs are following the ADD.
nr_premappedis therere unchanged, since it does not change the consistency between M-EPT and S-EPT.3. Unexpected zaps (such as
kvm_zap_gfn_range()).Side topic related to
kvm_zap_gfn_range(), theKVM_BUG_ON()invt_refresh_apicv_exec_ctrl()is flawed. Ifkvm_recalculate_apic_map()fails to allocate an optimized map, KVM will mark APICv as inhibited, i.e. the associatedWARN_ON_ONCE()is effectively user-triggerable.Easiest thing would be to mark the vCPU as dead (though we obviously need "KVM: Never clear
KVM_REQ_VM_DEADfrom a vCPU's requests" for that to be robust).arch/x86/kvm/vmx/main.c +2
-1@@ -719,7 +719,8 @@ static void vt_set_apic_access_page_addr(struct kvm_vcpu *vcpu)static void vt_refresh_apicv_exec_ctrl (struct kvm_vcpu *vcpu) {if (is_td_vcpu (vcpu)) { -KVM_BUG_ON (!kvm_vcpu_apicv_active (vcpu), vcpu->kvm); +if (!kvm_vcpu_apicv_active (vcpu)) +kvm_make_request (KVM_REQ_VM_DEAD , vcpu);return ; }Quoted 22 lines
Those zaps are currently just paranoid ones. Not found in any existing paths yet. i.e.,
We want to detect any future code or any missed code piecies, which invokeskvm_zap_gfn_range()(or maybe zaps under readmmu_lock).As those zaps do not necessarily hold
slot_lockor filemap lock, they may ocurr after installing M-EPT and before installing S-EPT.
As a result, the BLOCK fails andtdx_is_sept_zap_err_due_to_premap()returns true.
Decreasingnr_premappedhere to indicate the count of pages mapped in M-EPT but not in S-EPT decreases.TDH.MEM.PAGE.ADD after this zap can still succeed. If this occurs, the page will be mapped in S-EPT only. As KVM also decreases
nr_premappedafter a successful TDH.MEM.PAGE.ADD, thenr_premappedwill be <0 in the end. So, we will be able to detect those unexpected zaps.When userspace invokes
KVM_TDX_FINALIZE_VM,
=======
Thenr_premappedmust be 0 beforetdx_td_finalize()succeeds.Quoted 7 lines
The
nr_premappedcould be 0 if
(1) userspace invokesKVM_TDX_INIT_MEM_REGIONsas in a normal way.
(2) userspace never triggers anyKVM_TDX_INIT_MEM_REGION.
(3) userspace triggersKVM_TDX_INIT_MEM_REGIONbut zaps all initial memory regions.For (2)and(3),
KVM_TDX_FINALIZE_VMcan still succeed.Ya, we're in agreement on what can happen. I think all of the confusion was due to me forgetting that TDH.MEM.PAGE.ADD is what actually installes the leaf S-EPT entry.
Quoted 1 line
So, TD can still run with uninitialized memory.
No, the TD can never run with truly uninitialized memory. By "uninitialized", I mean memory that the guest can access and which has not been written to. Again, my confusion was due to thinking a page was already mapped into the guest, but awaiting TDH.MEM.PAGE.ADD to
Quoted 2 lines
Quoted 1 line
Side topic, why does KVM tolerate
tdh_mem_page_add()failure? IIUC, playingWe don't. It returns
-EBUSYor-EIOimmediately.But that _is_ tolerating failure, in the sense that KVM doesn't prevent further actions on the VM. Tolerating failure is fine in general, but in this case it leaves the MMU is left in a half-baked state.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
nice with
tdh_mem_page_add()failure necessitates both thetdx_is_sept_zap_err_due_to_premap()craziness and the check intdx_td_finalize()that all pending pages have been consumed.
tdx_is_sept_zap_err_due_to_premap()detects the error of BLOCK, which is caused by executing BLOCK before ADD.We need to make this situation impossible.
Currently this situation should be impossible already. If there're still missing ones, we can fix it (as you did above).
But this tdx_is_sept_zap_err_due_to_premap() check is just to detect if anything is still missing.
Or maybe directly KVM_BUG_ON() on that is also ok.
Quoted 23 lines
Quoted 12 lines
Quoted 1 line
What reasonable use case is there for gracefully handling
tdh_mem_page_add()failure?If
tdh_mem_page_add()fails, theKVM_TDX_INIT_MEM_REGIONjust fails.Quoted 3 lines
If there is a need to handle failure, I gotta imagine it's only for the
-EBUSYcase. And if it's only for-EBUSY, why can't that be handled by retrying intdx_vcpu_init_mem_region()? Iftdx_vcpu_init_mem_region()guarantees that allI analyzed the contention status of
tdh_mem_sept_add()at
https://lore.kernel.org/kvm/[email protected].As the userspace is expected to execute
KVM_TDX_INIT_MEM_REGIONin only one vCPU, returning-EBUSYinstead of retrying looks safer and easier.Quoted 6 lines
Quoted 4 lines
pages mapped into the S-EPT are ADDed, then it can assert that there are no pending pages when it completes (even if it "fails"), and similarly
tdx_td_finalize()can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
tdx_td_finalize()now just returns-EINVALin case ofnr_premappedbeing !0. KVM_BUG_ON/WARN_ON should be also ok.Ok, so I vaguely recall that I may have pushed back on using a scratch field in "struct
kvm_tdx" for temporary data (or maybe it was abusing vcpus[0] that I disliked?), but what we ended up with is far worse.
Quoted 7 lines
For all intents and purposes,
nr_premapped_is_ a temporary scratch field, but with access rules that are all but impossible to understand, e.g. there's practically zero chance anyone could suss out complications with "Unexpected zaps", and the existence of that super subtle edge case necessitates using an atomic because KVM can't strictly guarantee that access to the field is mutually exclusive. And that also means it's inherently racy, e.g. if a zap happens whiletdx_td_finalize()is checkingnr_premapped, what happens?
The tdx_td_finalize() takes slots_lock and you asserted those unexpected zaps at https://lore.kernel.org/all/[email protected].
Expected zaps can't occur during tdx_td_finalize checking nr_premapped either.
Quoted 4 lines
The real killer is that deferring TDH.MEM.PAGE.ADD and TDH.MR_EXTEND until after the map completes and
mmu_lockis dropped means that failure at that point leaves the TDP MMU in an inconsistent state, where the M-EPT has a present page but the S-EPT does not. Eww.
Eww... That's why there's nr_premapped.
And it's suggested by you, though you called it "Crazy idea"...
https://lore.kernel.org/kvm/[email protected]/
Quoted 4 lines
Note, in no way am I trying to blame anyone; quite the opposite, you've done an admirable job to get all of this landed. And I apologize if any of my past feedback led y'all down this road. I suspect my prefaulting idea really screwed things up; sorry :-(
It's ok :)
Quoted 9 lines
Back to the code, unless I'm missing yet another complication, I think the obvious fix to all of this is to pass the source page and metadata flags via a scratch field in "struct
kvm_tdx", and then do PAGE.ADD and MR.EXTEND intdx_sept_set_private_spte(). Then there is no need to keep track of pending pages, because the M-EPT and S-EPT are always consistent. E.g. if PAGE.ADD fails with-EBUSY, then KVM will naturally revert the M-EPT entry from FROZEN to !PRESENT. It also allows KVM toKVM_BUG_ON()MR.EXTEND failure, because it should be impossible for the S-EPT entry to be modified between PAGE.ADD and MR.EXTEND.
Quoted 3 lines
Diff on top below for feedback on the idea. A proper series for this would simply replace several of the patches, e.g. asserting that
slots_lockis held ontdx_is_sept_zap_err_due_to_premap()is wrong.
Looks it's similar to the implementation in v19?
https://lore.kernel.org/all/bbac4998cfb34da496646491038b03f501964cbd.1708933498.git.isaku.yamahata@intel.com/
Quoted 255 lines
--- arch/x86/kvm/vmx/tdx.c | 157 +++++++++++++++++------------------------ arch/x86/kvm/vmx/tdx.h | 11 +-- 2 files changed, 70 insertions(+), 98 deletions(-)arch/x86/kvm/vmx/tdx.c +64
-93@@ -1586,6 +1586,56 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } + +struct kvm_tdx_page_add { +struct page *src; +unsigned long flags; +}; + +static int tdx_mem_page_add (struct kvm *kvm,gfn_t gfn,enum pg_level level, +kvm_pfn_t pfn) +{ +struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); +u64 err, entry, level_state; +gpa_t gpa =gfn_to_gpa (gfn); +int i; + +lockdep_assert_held (&kvm->slots_lock); + +if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm) || +KVM_BUG_ON (!kvm_tdx->page_add, kvm)) +return -EIO ; + + err =tdh_mem_page_add (&kvm_tdx->td, gpa,pfn_to_page (pfn), + kvm_tdx->page_add->src, &entry, &level_state); +if (unlikely (tdx_operand_busy (err))) +return -EBUSY ; + +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error_2 (TDH_MEM_PAGE_ADD , err, entry, level_state); +return -EIO ; + } + +if (!(kvm_tdx->page_add->flags &KVM_TDX_MEASURE_MEMORY_REGION )) +return 0 ; + +/* +* Extend the measurement while holding mmu_lock to ensure MR.EXTEND +* can't fail, e.g. due to the S-EPT entry being zapped after PAGE.ADD. +* Note! If extended the measurement fails, bug the VM, but do NOT +* return an error, as mapping the page in the S-EPT succeeded and so +* needs to be tracked in KVM's mirror page tables. +*/ +for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) { + err =tdh_mr_extend (&kvm_tdx->td, gpa + i, &entry, &level_state); +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error_2 (TDH_MR_EXTEND , err, entry, level_state); +break ; + } + } +return 0 ; +} +static int tdx_mem_page_aug (struct kvm *kvm,gfn_t gfn,enum pg_level level,kvm_pfn_t pfn) {@@ -1627,21 +1677,11 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,/* * If the TD isn't finalized/runnable, then userspace is initializing -* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of -* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD -* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks -* the counter to ensure all mapped pages have been added to the image, -* to prevent running the TD with uninitialized memory. +* the VM image via KVM_TDX_INIT_MEM_REGION. Add the page to the TD, +* and optionally extend the measurement with the page contents. */ -if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) { -lockdep_assert_held (&kvm->slots_lock); - -if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm)) -return -EIO ; - - kvm_tdx->nr_pending_tdh_mem_page_adds++; -return 0 ; - } +if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) +return tdx_mem_page_add (kvm, gfn, level, pfn);return tdx_mem_page_aug (kvm, gfn, level, pfn); }@@ -1716,39 +1756,6 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,return 0 ; } -/* -* Check if the error returned from a SEPT zap SEAMCALL is due to that a page is -* mapped by KVM_TDX_INIT_MEM_REGION without tdh_mem_page_add() being called -* successfully. -* -* Since tdh_mem_sept_add() must have been invoked successfully before a -* non-leaf entry present in the mirrored page table, the SEPT ZAP related -* SEAMCALLs should not encounter err TDX_EPT_WALK_FAILED. They should instead -* find TDX_EPT_ENTRY_STATE_INCORRECT due to an empty leaf entry found in the -* SEPT. -* -* Further check if the returned entry from SEPT walking is with RWX permissions -* to filter out anything unexpected. -* -* Note: @level is pg_level, not the tdx_level. The tdx_level extracted from -* level_state returned from a SEAMCALL error is the same as that passed into -* the SEAMCALL. -*/ -static int tdx_is_sept_zap_err_due_to_premap (struct kvm_tdx *kvm_tdx,u64 err, -u64 entry,int level) -{ -if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE ) -return false; - -if (err != (TDX_EPT_ENTRY_STATE_INCORRECT |TDX_OPERAND_ID_RCX )) -return false; - -if ((is_last_spte (entry, level) && (entry &VMX_EPT_RWX_MASK ))) -return false; - -return true; -} -static int tdx_sept_zap_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {@@ -1768,15 +1775,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn, err =tdh_mem_range_block (&kvm_tdx->td, gpa, tdx_level, &entry, &level_state);tdx_no_vcpus_enter_stop (kvm); } -if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) { -lockdep_assert_held (&kvm->slots_lock); - -if (KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm)) -return -EIO ; - -return 0 ; - } -if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_RANGE_BLOCK , err, entry, level_state);return -EIO ;@@ -2842,12 +2840,6 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd)if (!is_hkid_assigned (kvm_tdx) || kvm_tdx->state ==TD_STATE_RUNNABLE )return -EINVAL ; -/* -* Pages are pending for KVM_TDX_INIT_MEM_REGION to issue -* TDH.MEM.PAGE.ADD(). -*/ -if (kvm_tdx->nr_pending_tdh_mem_page_adds) -return -EINVAL ; cmd->hw_error =tdh_mr_finalize (&kvm_tdx->td);if (tdx_operand_busy (cmd->hw_error))@@ -3131,50 +3123,29 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, {struct tdx_gmem_post_populate_arg *arg = _arg;struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm); -u64 err, entry, level_state; -gpa_t gpa =gfn_to_gpa (gfn); -struct page *src_page; -int ret, i; +struct kvm_tdx_page_add page_add = { + .flags = arg->flags, + }; +int ret; -lockdep_assert_held (&kvm->slots_lock); +if (KVM_BUG_ON (kvm_tdx->page_add, kvm)) +return -EIO ;/* * Get the source page if it has been faulted in. Return failure if the * source page has been swapped out or unmapped in primary memory. */ - ret =get_user_pages_fast ((unsigned long )src,1 ,0 , &src_page); + ret =get_user_pages_fast ((unsigned long )src,1 ,0 , &page_add.src);if (ret <0 )return ret;if (ret !=1 )return -ENOMEM ; + kvm_tdx->page_add = &page_add; ret =kvm_tdp_mmu_map_private_pfn (arg->vcpu, gfn, pfn); -if (ret <0 ) - goto out; + kvm_tdx->page_add =NULL ; - ret =0 ; - err =tdh_mem_page_add (&kvm_tdx->td, gpa,pfn_to_page (pfn), - src_page, &entry, &level_state); -if (err) { - ret =unlikely (tdx_operand_busy (err)) ? -EBUSY : -EIO ; - goto out; - } - -KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm); - -if (arg->flags &KVM_TDX_MEASURE_MEMORY_REGION ) { -for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) { - err =tdh_mr_extend (&kvm_tdx->td, gpa + i, &entry, - &level_state); -if (err) { - ret = -EIO ; -break ; - } - } - } - -out: -put_page (src_page); +put_page (page_add.src);return ret; }arch/x86/kvm/vmx/tdx.h +6
-5@@ -21,6 +21,8 @@ enum kvm_tdx_state { TD_STATE_RUNNABLE, }; +struct kvm_tdx_add_page ; +struct kvm_tdx {struct kvm kvm;@@ -37,12 +39,11 @@ struct kvm_tdx {struct tdx_td td;/* -* The number of pages that KVM_TDX_INIT_MEM_REGION has mapped into the -* S-EPT, but not yet initialized via TDH.MEM.PAGE_ADD. Used to sanity -* check adding pages to the image, and to ensure that all pages have -* been initialized before finalizing the TD. +* Scratch structure used to pass the source page and metadata flags to +* tdx_mem_page_add. Protected by slots_lock, and non-NULL only when +* mapping a private pfn via tdx_gmem_post_populate(). */ -unsigned long nr_pending_tdh_mem_page_adds; +struct kvm_tdx_page_add *page_add;/* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do base-commit:
7c7a3893b102bdeb4826f7140280b7b16081b385--
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
Quoted 5 lines · Yan Zhao
Quoted 4 lines · Sean Christopherson
Quoted 3 lines · Yan Zhao
Quoted 2 lines · Sean Christopherson
Quoted 1 line · Yan Zhao
On Tue, Aug 26, 2025 at 05:05:19PM -0700, Sean Christopherson wrote:
...
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Quoted 1 line
Side topic, why does KVM tolerate
tdh_mem_page_add()failure? IIUC, playingWe don't. It returns
-EBUSYor-EIOimmediately.But that _is_ tolerating failure, in the sense that KVM doesn't prevent further actions on the VM. Tolerating failure is fine in general, but in this case it leaves the MMU is left in a half-baked state.
Yes, but nr_premapped will not be decreased on tdh_mem_page_add() failure.
So we rely on nr_premapped to disallow the TD from running eventually.
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Sean ChristophersonQuoted 4 lines · Sean Christopherson
Quoted 3 lines · Yan Zhao
Quoted 2 lines · Sean Christopherson
arch/x86/kvm/vmx/tdx.c +0
-0@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;
[snip]
Quoted 47 lines
Quoted 12 lines
Quoted 12 lines
arch/x86/kvm/vmx/tdx.c +4
-4/* -* Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching -* barrier in tdx_td_finalize(). +* Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() +* before kvm_tdx->state. Userspace must not be allowed to pre-fault +* arbitrary memory until the initial memory image is finalized. Pairs +* with the smp_wmb() in tdx_td_finalize(). */ smp_rmb (); -if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE )) -return tdx_mem_page_aug (kvm, gfn, level, pfn);Quoted 9 lines
arch/x86/kvm/vmx/tdx.c static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1634
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +7
-1-return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +/* +* If the TD isn't finalized/runnable, then userspace is initializing +* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of +* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD +* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks +* the counter to ensure all mapped pages have been added to the image, +* to prevent running the TD with uninitialized memory. To prevent the mismatch between mirror EPT and the S-EPT?
No? Because KVM bumps the count when installing the S-EPT and decrements it on AUG, so I don't see how
nr_premappedguards against M-EPT vs. S-EPT issues?Quoted 4 lines
e.g., Before
KVM_TDX_FINALIZE_VM, if userspace performs a zap after the TDH.MEM.PAGE.ADD, the page will be removed from the S-EPT. The count ofnr_premappedwill not change after the successful TDH.MEM.RANGE.BLOCK and TDH.MEM.PAGE.REMOVE.Eww. It would be nice to close that hole, but I suppose it's futile, e.g. the underlying problem is unexpectedly removing pages from the initial, whether the VMM is doing stupid things before vs. after FINALIZE doesn't really matter.
Quoted 1 line
As a result, the TD will still run with uninitialized memory.
No? Because BLOCK+REMOVE means there are no valid S-EPT mappings. There's a "hole" that the guest might not expect, but that hole will trigger an EPT violation and only get "filled" if the guest explicitly accepts an AUG'd page.
Side topic, why does KVM tolerate
tdh_mem_page_add()failure? IIUC, playing nice withtdh_mem_page_add()failure necessitates both thetdx_is_sept_zap_err_due_to_premap()craziness and the check intdx_td_finalize()that all pending pages have been consumed.What reasonable use case is there for gracefully handling
tdh_mem_page_add()failure?
Quoted 7 lines
If there is a need to handle failure, I gotta imagine it's only for the
-EBUSYcase. And if it's only for-EBUSY, why can't that be handled by retrying intdx_vcpu_init_mem_region()? Iftdx_vcpu_init_mem_region()guarantees that all pages mapped into the S-EPT are ADDed, then it can assert that there are no pending pages when it completes (even if it "fails"), and similarlytdx_td_finalize()can KVM_BUG_ON/WARN_ON the number of pending pages being non-zero.
Ah just reading this... yea I'm wondering the same thing.
Ira
[snip]
Re: [RFC PATCH 09/12] KVM: TDX: Fold tdx_mem_page_record_premap_cnt() into its sole caller
In reply to Yan ZhaoQuoted 1 line · Yan Zhao
On Tue, Aug 26, 2025 at 05:05:19PM -0700, Sean Christopherson wrote:
[snip]
Quoted 25 lines
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c +4
-3@@ -1641,14 +1618,30 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;/* -* Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching -* barrier in tdx_td_finalize(). +* Ensure pre_fault_allowed is read by kvm_arch_vcpu_pre_fault_memory() +* before kvm_tdx->state. Userspace must not be allowed to pre-fault +* arbitrary memory until the initial memory image is finalized. Pairs +* with the smp_wmb() in tdx_td_finalize(). */ smp_rmb (); -if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE ))arch/x86/kvm/vmx/tdx.c static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1634
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-2-return tdx_mem_page_aug (kvm, gfn, level, pfn); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +/* +* If the TD isn't finalized/runnable, then userspace is initializing +* the VM image via KVM_TDX_INIT_MEM_REGION. Increment the number of +* pages that need to be initialized via TDH.MEM.PAGE.ADD (PAGE.ADD +* requires a pre-existing S-EPT mapping). KVM_TDX_FINALIZE_VM checks +* the counter to ensure all mapped pages have been added to the image, +* to prevent running the TD with uninitialized memory. To prevent the mismatch between mirror EPT and the S-EPT?
Quoted 6 lines
e.g., Before
KVM_TDX_FINALIZE_VM,
if userspace performs a zap after the TDH.MEM.PAGE.ADD, the page will be removed from the S-EPT. The count ofnr_premappedwill not change after the successful TDH.MEM.RANGE.BLOCK and TDH.MEM.PAGE.REMOVE.As a result, the TD will still run with uninitialized memory.
I'm wondering if we are trying to over-architect this.
Should we even allow KVM_TDX_FINALIZE_VM to race with KVM_TDX_INIT_MEM_REGION? What is the use case for that?
It seems a basic sanity check/KVM_BUG_ON would suffice to tell the user; Don't start adding memory dynamically until yall have finalized the VM.
Ira
Sean Christopherson
maintainerRFC PATCH 10/12 KVM: TDX: Assert that slots_lock is held when nr_premapped is accessed
In reply to Sean ChristophersonAssert that slots_lock is held when the TDX codes accesses the number of premapped pfns, as KVM relies on calls to tdx_vcpu_init_mem_region() being serialized to prevent double-population of gmem and false negatives on the consumption of a "premapped" pfn. In addition to helping document how the TDX code works, this will allow converting "nr_premapped" to a non-atomic variable, as all usage asserts that slots_lock is held.
--- arch/x86/kvm/vmx/tdx.c | 6 ++++++ 1 file changed, 6 insertions(+)
arch/x86/kvm/vmx/tdx.c +6 -0
@@ -1634,6 +1634,8 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* to prevent running the TD with uninitialized memory. */ if (unlikely (kvm_tdx->state !=TD_STATE_RUNNABLE )) { +lockdep_assert_held (&kvm->slots_lock); +if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EIO ;
@@ -1767,6 +1769,8 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,tdx_no_vcpus_enter_stop (kvm); }if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) { +lockdep_assert_held (&kvm->slots_lock); +if (KVM_BUG_ON (atomic64_dec_return (&kvm_tdx->nr_premapped) <0 , kvm))return -EIO ;
@@ -3132,6 +3136,8 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,struct page *src_page;int ret, i; +lockdep_assert_held (&kvm->slots_lock); +/* * Get the source page if it has been faulted in. Return failure if the * source page has been swapped out or unmapped in primary memory.
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Sean Christopherson
maintainerRFC PATCH 11/12 KVM: TDX: Track nr_premapped as an "unsigned long", not an "atomic64_t"
In reply to Sean ChristophersonTrack the number of premapped pfns as a non-atomic variable as all usage is guarded by slots_lock, and KVM now asserts as much. Note, slots_lock has always effectively guarded nr_premapped since TDX support landed, the use of an atomic64_t was likely a leftover from development that was never cleaned up.
--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- arch/x86/kvm/vmx/tdx.h | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -4
@@ -1639,7 +1639,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EIO ; -atomic64_inc (&kvm_tdx->nr_premapped); + kvm_tdx->nr_premapped++;return 0 ; }
@@ -1771,7 +1771,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) {lockdep_assert_held (&kvm->slots_lock); -if (KVM_BUG_ON (atomic64_dec_return (&kvm_tdx->nr_premapped) <0 , kvm)) +if (KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm))return -EIO ;return 0 ;
@@ -2846,7 +2846,7 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd)* Pages are pending for KVM_TDX_INIT_MEM_REGION to issue * TDH.MEM.PAGE.ADD(). */ -if (atomic64_read (&kvm_tdx->nr_premapped)) +if (kvm_tdx->nr_premapped)return -EINVAL ; cmd->hw_error =tdh_mr_finalize (&kvm_tdx->td);
@@ -3160,7 +3160,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, goto out; } -KVM_BUG_ON (atomic64_dec_return (&kvm_tdx->nr_premapped) <0 , kvm); +KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm);if (arg->flags &KVM_TDX_MEASURE_MEMORY_REGION ) {for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) {
arch/x86/kvm/vmx/tdx.h +1 -1
@@ -37,7 +37,7 @@ struct kvm_tdx {struct tdx_td td;/* For KVM_TDX_INIT_MEM_REGION. */ -atomic64_t nr_premapped; +unsigned long nr_premapped;/* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 11/12] KVM: TDX: Track nr_premapped as an "unsigned long", not an "atomic64_t"
Quoted 58 lines · Sean Christopherson
Track the number of premapped pfns as a non-atomic variable as all usage is guarded by
slots_lock, and KVM now asserts as much. Note,slots_lockhas always effectively guardednr_premappedsince TDX support landed, the use of anatomic64_twas likely a leftover from development that was never cleaned up.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- arch/x86/kvm/vmx/tdx.h | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/tdx.c +2
-2@@ -1639,7 +1639,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EIO ; -atomic64_inc (&kvm_tdx->nr_premapped); + kvm_tdx->nr_premapped++;return 0 ; }@@ -1771,7 +1771,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) {lockdep_assert_held (&kvm->slots_lock); -if (KVM_BUG_ON (atomic64_dec_return (&kvm_tdx->nr_premapped) <0 , kvm)) +if (KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm))return -EIO ;arch/x86/kvm/vmx/tdx.c +2
-2return 0 ;@@ -2846,7 +2846,7 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd)* Pages are pending for KVM_TDX_INIT_MEM_REGION to issue * TDH.MEM.PAGE.ADD(). */ -if (atomic64_read (&kvm_tdx->nr_premapped)) +if (kvm_tdx->nr_premapped)return -EINVAL ; cmd->hw_error =tdh_mr_finalize (&kvm_tdx->td);@@ -3160,7 +3160,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, goto out; } -KVM_BUG_ON (atomic64_dec_return (&kvm_tdx->nr_premapped) <0 , kvm); +KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm);if (arg->flags &KVM_TDX_MEASURE_MEMORY_REGION ) {for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) {arch/x86/kvm/vmx/tdx.h +0
-0
arch/x86/kvm/vmx/tdx.h struct kvm_tdx { · line 40
Quoted 5 lines
arch/x86/kvm/vmx/tdx.h +1
-1@@ -37,7 +37,7 @@ struct kvm_tdx {struct tdx_td td;/* For KVM_TDX_INIT_MEM_REGION. */ -atomic64_t nr_premapped; +unsigned long nr_premapped;
Due to the comparison with < 0, the type should be "s64" or "signed long"?
Quoted 5 lines
arch/x86/kvm/vmx/tdx.h +0
-0/* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do Signature · 1 line
2.51.0.268.g9569e192d0-goog
Sean Christopherson
maintainerRFC PATCH 12/12 KVM: TDX: Rename nr_premapped to nr_pending_tdh_mem_page_adds
In reply to Sean ChristophersonRename "nr_premapped" to an asurdly verbose "nr_pending_tdh_mem_page_adds" to make it explicitly clear what the counter tracks. "pre-map" is far too similar to "pre-fault", especially since tdx_sept_set_private_spte() deals with both "pre_fault_allowed" and the counter.
No functional change intended.
--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- arch/x86/kvm/vmx/tdx.h | 9 +++++++-- 2 files changed, 11 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -4
@@ -1639,7 +1639,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EIO ; - kvm_tdx->nr_premapped++; + kvm_tdx->nr_pending_tdh_mem_page_adds++;return 0 ; }
@@ -1771,7 +1771,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) {lockdep_assert_held (&kvm->slots_lock); -if (KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm)) +if (KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm))return -EIO ;return 0 ;
@@ -2846,7 +2846,7 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd)* Pages are pending for KVM_TDX_INIT_MEM_REGION to issue * TDH.MEM.PAGE.ADD(). */ -if (kvm_tdx->nr_premapped) +if (kvm_tdx->nr_pending_tdh_mem_page_adds)return -EINVAL ; cmd->hw_error =tdh_mr_finalize (&kvm_tdx->td);
@@ -3160,7 +3160,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, goto out; } -KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm); +KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm);if (arg->flags &KVM_TDX_MEASURE_MEMORY_REGION ) {for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) {
arch/x86/kvm/vmx/tdx.h +7 -2
@@ -36,8 +36,13 @@ struct kvm_tdx {struct tdx_td td; -/* For KVM_TDX_INIT_MEM_REGION. */ -unsigned long nr_premapped; +/* +* The number of pages that KVM_TDX_INIT_MEM_REGION has mapped into the +* S-EPT, but not yet initialized via TDH.MEM.PAGE_ADD. Used to sanity +* check adding pages to the image, and to ensure that all pages have +* been initialized before finalizing the TD. +*/ +unsigned long nr_pending_tdh_mem_page_adds;/* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do
Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 12/12] KVM: TDX: Rename nr_premapped to nr_pending_tdh_mem_page_adds
Quoted 59 lines · Sean Christopherson
Rename "
nr_premapped" to an asurdly verbose "nr_pending_tdh_mem_page_adds" to make it explicitly clear what the counter tracks. "pre-map" is far too similar to "pre-fault", especially sincetdx_sept_set_private_spte()deals with both "pre_fault_allowed" and the counter.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 8 ++++---- arch/x86/kvm/vmx/tdx.h | 9 +++++++-- 2 files changed, 11 insertions(+), 6 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-4@@ -1639,7 +1639,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EIO ; - kvm_tdx->nr_premapped++; + kvm_tdx->nr_pending_tdh_mem_page_adds++;return 0 ; }@@ -1771,7 +1771,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level)) {lockdep_assert_held (&kvm->slots_lock); -if (KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm)) +if (KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm))return -EIO ;return 0 ;@@ -2846,7 +2846,7 @@ static int tdx_td_finalize(struct kvm *kvm, struct kvm_tdx_cmd *cmd)* Pages are pending for KVM_TDX_INIT_MEM_REGION to issue * TDH.MEM.PAGE.ADD(). */ -if (kvm_tdx->nr_premapped) +if (kvm_tdx->nr_pending_tdh_mem_page_adds)return -EINVAL ; cmd->hw_error =tdh_mr_finalize (&kvm_tdx->td);@@ -3160,7 +3160,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, goto out; } -KVM_BUG_ON (--kvm_tdx->nr_premapped <0 , kvm); +KVM_BUG_ON (--kvm_tdx->nr_pending_tdh_mem_page_adds <0 , kvm);if (arg->flags &KVM_TDX_MEASURE_MEMORY_REGION ) {for (i =0 ; i <PAGE_SIZE ; i +=TDX_EXTENDMR_CHUNKSIZE ) {arch/x86/kvm/vmx/tdx.h +0
-0@@ -36,8 +36,13 @@ struct kvm_tdx {
arch/x86/kvm/vmx/tdx.h struct kvm_tdx { · line 44
Quoted 10 lines
arch/x86/kvm/vmx/tdx.h +6
-2struct tdx_td td; -/* For KVM_TDX_INIT_MEM_REGION. */ -unsigned long nr_premapped; +/* +* The number of pages that KVM_TDX_INIT_MEM_REGION has mapped into the +* S-EPT, but not yet initialized via TDH.MEM.PAGE_ADD. Used to sanity +* check adding pages to the image, and to ensure that all pages have +* been initialized before finalizing the TD. +*/
To ensure the consistency between mirror EPT and S-EPT? (as in
https://lore.kernel.org/all/[email protected]).
tdx_td_finalize() holds slots_lock, so it can't run in-between a KVM_TDX_INIT_MEM_REGION.
Quoted 7 lines
arch/x86/kvm/vmx/tdx.h +1
-0+unsigned long nr_pending_tdh_mem_page_adds;/* * Prevent vCPUs from TD entry to ensure SEPT zap related SEAMCALLs do Signature · 1 line
2.51.0.268.g9569e192d0-goog
Re: [RFC PATCH 12/12] KVM: TDX: Rename nr_premapped to nr_pending_tdh_mem_page_adds
In reply to Sean ChristophersonQuoted 8 lines · Sean Christopherson
Rename "
nr_premapped" to an asurdly verbose "nr_pending_tdh_mem_page_adds" to make it explicitly clear what the counter tracks. "pre-map" is far too similar to "pre-fault", especially sincetdx_sept_set_private_spte()deals with both "pre_fault_allowed" and the counter.No functional change intended.
Signed-off-by Sean Christopherson <[email protected]>
[snip]
Quoted 21 lines · Sean Christopherson
This is a largely untested series to do most of what was discussed in the thread regarding locking issues between gmem and TDX's post-populate hook[*], with more than a few side quests thrown in as I was navigating through the code to try to figure out how best to eliminate the
copy_from_user()fromsev_gmem_post_populate(), which has the same locking problem (copying from a userspace address can fault and in theory trigger the same problematic path, I think).Notably absent is the extraction of
copy_from_user()fromsev_gmem_post_populate()tokvm_gmem_populate(). I've had this on my todo list for a few weeks now, and haven't been able to focus on it for long enough to get something hammered out, and with KVM Forum on the horizon, I don't anticipate getting 'round to it within the next month (if not much longer).The thing that stymied me is what to do if
snp_launch_update()is passed in a huge batch of pages. I waffled between doing a slow one-at-a-time approach and a batched approached, and got especially stuck when trying to remember and/or figure out how that handling would interact with hugepage support in SNP in particular.
Quoted 3 lines
If anyone wants to tackle that project, the one thing change I definitely think we should do is change the
post_populate()callback to operate on exactly one page.
Not sure if I understand it correctly.
Do you mean something like the tdx_gmem_post_populate_4k() in
https://lore.kernel.org/all/[email protected], or invoking hugepage_set_guest_inhibit() in the post_populate() callback?
Quoted 38 lines
KVM_SEV_SNP_LAUNCH_UPDATEallows for partial progress,
i.e. KVM's ABI doesn't require it to unwind a batch if adding a page fails. If we take advantage of that, thensev_gmem_post_populate()will be a bit simpler (though I wouldn't go so far as to call it "simple").RFC as this is compile tested only (mostly due to lack of access to a TDX capable system, but also due to lack of cycles).
[*] http://lore.kernel.org/all/[email protected]
Sean Christopherson (12):
KVM: TDX: Drop PROVE_MMU=y sanity check on to-be-populated mappings
KVM: x86/mmu: Add dedicated API to mapguest_memfdpfn into TDP MMU Revert "KVM: x86/tdp_mmu: Add a helper function to walk down the TDP
MMU"
KVM: x86/mmu: Renamekvm_tdp_map_page()tokvm_tdp_prefault_page()
KVM: TDX: Drop superfluous page pinning in S-EPT management
KVM: TDX: Return-EIO, not-EINVAL, on aKVM_BUG_ON()condition
KVM: TDX: Avoid a double-KVM_BUG_ON() intdx_sept_zap_private_spte()
KVM: TDX: Useatomic64_dec_return()instead of a poor equivalent
KVM: TDX: Foldtdx_mem_page_record_premap_cnt()into its sole caller
KVM: TDX: Assert thatslots_lockis held whennr_premappedis accessed
KVM: TDX: Tracknr_premappedas an "unsigned long", not an"atomic64_t"KVM: TDX: Rename
nr_premappedtonr_pending_tdh_mem_page_addsarch/x86/kvm/mmu.h | 3 +- arch/x86/kvm/mmu/mmu.c | 66 ++++++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 37 ++--------- arch/x86/kvm/vmx/tdx.c | 123 +++++++++++++------------------------ arch/x86/kvm/vmx/tdx.h | 9 ++- 5 files changed, 117 insertions(+), 121 deletions(-)base-commit:
196d9e72c4b0bd68b74a4ec7f52d248f37d0f030Signature · 1 line
2.51.0.268.g9569e192d0-goog
Edgecombe, Rick P
maintainerQuoted 2 lines · Sean Christopherson
RFC as this is compile tested only (mostly due to lack of access to a TDX capable system, but also due to lack of cycles).
Let us know how we could best help with this. The series fails the tests because of the page size issue Yan pointed. We could just review and test a v2, or if you want us to pull together the feedback, test the result, and repost please let us know. I think either should work from our end.
I suspect Vishal could hook you up with a TDX machine. But if you need any setup help there too, please shout.
Sean Christopherson
maintainerQuoted 8 lines · Rick P
Quoted 2 lines · Sean Christopherson
RFC as this is compile tested only (mostly due to lack of access to a TDX capable system, but also due to lack of cycles).
Let us know how we could best help with this. The series fails the tests because of the page size issue Yan pointed. We could just review and test a v2, or if you want us to pull together the feedback, test the result, and repost please let us know. I think either should work from our end.
I'll post a v2, it's going to look quite different.
Quoted 2 lines
I suspect Vishal could hook you up with a TDX machine. But if you need any setup help there too, please shout.
Oh, he can, I just haven't crossed that testing bridge yet (ditto for SNP). I'll do so someday, but for now I'll abuse your generosity and throw noodles at ya :-)