KVM: TDX huge page support for private memory
Subsystems (16)
cover KVM: TDX huge page support for private memory
This is an RFC series to support huge pages in TDX. It's an evolution of the previous patches from Isaku [0]. (Please find the main changes to [0] at a later section).
As the series of enabling guest_memfd to support 1GB huge page with in-place conversion [1] is still under development, we temporarily based the TDX work on top of the series from Michael Roth that enables basic 2M guest_memfd support without in-place conversion[2]. The goal is to have an early review and discussion of the TDX huge page work (including changes to KVM core MMU and the TDX specific code), which should remain stable, with only minor adjustments, regardless the changes coming in guest_memfd.
The series is currently focused on supporting 2MB huge pages only.
Tip folks, there are some SEAMCALL wrapper changes in this series, but we need to have some discussion on the KVM side to figure out what it needs still. Please feel free to ignore it for now.
Design
======guest_memfd
-----------
TDX huge page support has a basic assumption to guest_memfd: guest_memfd allocates private huge pages whenever alignment of GFN/index, range size and the consistency of page attributes allow.
Patch 01 (based on [2]) in this RFC acts as glue code to ensure this assumption is met for TDX. It can be absorbed into any future guest_memfd series (e.g., future in-place conversion series) in any form.
TDX interacts with guest_memfd through interfaces kvm_gmem_populate() and kvm_gmem_get_pfn(), obtaining the allocated page and its order.
The remaining TDX code should remain stable despite future changes in guest_memfd.
Basic huge page mapping/unmapping
---------------------------------
- TD build time
This series enforces that all private mappings be 4KB during the TD build phase, due to the TDX module's requirement that tdh_mem_page_add(), the SEAMCALL for adding private pages during TD build time, only supports 4KB mappings. Enforcing 4KB mappings also simplifies the implementation of code for TD build time, by eliminating the need to consider merging or splitting in the mirror page table during TD build time.
The underlying pages allocated from guest_memfd during TD build time phase can still be large, allowing for potential merging into 2MB mappings once the TD is running.
- TD runtime
This series allows a private fault's max_level to be 2MB after TD is running. KVM core MMU will map/unmap 2MB mappings in the mirror page table according to a fault's goal_level as what're done for normal VMs. Changes in the mirror page table are then propagated to the S-EPT.
For transitions from non-present to huge leaf in the mirror page table, hook set_external_spte is invoked, leading to the execution of tdh_mem_page_aug() to install a huge leaf in the S-EPT.
Conversely, during transitions from a huge leaf to non-present, the remove_external_spte hook is invoked to execute SEAMCALLs that remove the huge leaf from the S-EPT.
(For transitions from huge leaf to non-leaf, or from non-leaf to huge
leaf, SPTE splitting/merging will be triggered. More details are in later sections.)
- Specify fault max_level
In the TDP MMU, a fault's max_level is initially set to the 1GB level for x86. KVM then updates the fault's max_level by determining the lowest order among fault->max_level, the order of the allocated private page, and the TDX-specified max_level from hook private_max_mapping_level. For TDX, a private fault's req_level, and goal_level finally equal to the fault's max_level as TDX platforms do not have the flaw for NX huge page.
So, if TDX has specific requirements to influence a fault's goal_level for private memory (e.g., if it knows an EPT violation is caused by a TD's ACCEPT operation, mapping at the ACCEPT's level is preferred), this can be achieved either by affecting the initial value of fault->max_level or through the private_max_mapping_level hook.
The former approach requires more changes in the KVM core (e.g., by using some bits in the error_code passed to kvm_mmu_page_fault() and having KVM check for them). This RFC opts for the latter, simpler method, using the private_max_mapping_level hook.
Page splitting (page demotion)
------------------------------
Page splitting occurs in two paths:
(a) with exclusive kvm->mmu_lock, triggered by zapping operations,
For normal VMs, if zapping a narrow region that would need to split a huge page, KVM can simply zap the surrounding GFNs rather than splitting a huge page. The pages can then be faulted back in, where KVM can handle mapping them at a 4KB level.
The reason why TDX can't use the normal VM solution is that zapping private memory that is accepted cannot easily be re-faulted, since it can only be re-faulted as unaccepted. So KVM will have to sometimes do the page splitting as part of the zapping operations.
These zapping operations can occur for few reasons:
1. VM teardown.
2. Memslot removal.
3. Conversion of private pages to shared.
4. Userspace does a hole punch to guest_memfd for some reason.
For case 1 and 2, splitting before zapping is unnecessary because either the entire range will be zapped or huge pages do not span memslots.
Case 3 or case 4 requires splitting, which is also followed by a backend page splitting in guest_memfd.
(b) with shared kvm->mmu_lock, triggered by fault.
Splitting in this path is not accompanied by a backend page splitting (since backend page splitting necessitates a splitting and zapping
operation in the former path). It is triggered when KVM finds that a
non-leaf entry is replacing a huge entry in the fault path, which is usually caused by vCPUs' concurrent ACCEPT operations at different levels.
This series simply ignores the splitting request in the fault path to avoid unnecessary bounces between levels. The vCPU that performs ACCEPT at a lower level would finally figures out the page has been accepted at a higher level by another vCPU.
A rare case that could lead to splitting in the fault path is when a TD is configured to receive #VE and accesses memory before the ACCEPT operation. By the time a vCPU accesses a private GFN, due to the lack of any guest preferred level, KVM could create a mapping at 2MB level. If the TD then only performs the ACCEPT operation at 4KB level, splitting in the fault path will be triggered. However, this is not regarded as a typical use case, as usually TD always accepts pages in the order from 1GB->2MB->4KB. The worst outcome to ignore the resulting splitting request is an endless EPT violation. This would not happen for a Linux guest, which does not expect any #VE.
- Splitting for private-to-shared conversion or punch hole Splitting of a huge mapping requires the allocation of page table page and the corresponding shadow structures. This memory allocation can fail. So, while the zapping operations in the two scenarios don't have an understanding of failure, the overall operations do. Therefore, the RFC introduces a separate step kvm_split_boundary_leafs() to split huge mappings ahead of the zapping operation.
Patches 16-17 implement this change. As noted in the patch log, the downside of the current approach is that although kvm_split_boundary_leafs() is invoked before kvm_unmap_gfn_range() for each GFN range, the entire zapping range may consist of several GFN ranges. If an out-of-memory error occurs during the splitting of a GFN range, some previous GFN ranges may have been successfully split and zapped, even though their page attributes remain unchanged due to the splitting failure. This may not be a significant issue, as the user can retry the ioctl to split and zap the full range. However, if it becomes problematic, further modifications to invoke kvm_unmap_gfn_range() after executing kvm_mmu_invalidate_range_add() and kvm_split_boundary_leafs() for all GFN ranges could address the problem.
Alternatively, a possible solution could be pre-allocating sufficiently large splitting caches at the start of the private-to-shared conversion or hole punch process. The downside is that this may allocate more memory than necessary and require more code changes.
- The full call stack for huge page splitting
With exclusive kvm->mmu_lock,
kvm_vm_set_mem_attributes/kvm_gmem_punch_hole
|kvm_split_boundary_leafs
| |kvm_tdp_mmu_gfn_range_split_boundary
| |tdp_mmu_split_boundary_leafs
| |tdp_mmu_alloc_sp_for_split
| |tdp_mmu_split_huge_page
| |tdp_mmu_link_sp
| |tdp_mmu_iter_set_spte
| |tdp_mmu_set_spte
| |split_external_spt
| |kvm_x86_split_external_spt
| | BLOCK, TRACK, DEMOTION
|kvm_mmu_unmap_gfn_range With shared kvm->mmu_lock,
kvm_tdp_mmu_map
|tdp_mmu_alloc_sp
|kvm_mmu_alloc_external_spt
|tdp_mmu_split_huge_page
|tdp_mmu_link_sp
|tdp_mmu_set_spte_atomic
|__tdp_mmu_set_spte_atomic
|set_external_spte_present
|split_external_spt
|kvm_x86_split_external_spt- Handle busy & errors
Splitting huge mappings in S-EPT requires to execute tdh_mem_range_block(), tdh_mem_track(), kicking off vCPUs, tdh_mem_page_demote() in sequence.
Possible errors during the process include TDX_OPERAND_BUSY or TDX_INTERRUPTED_RESTARTABLE.
With exclusive kvm->mmu_lock, TDX_OPERAND_BUSY can be handled similarly to removing a private page, i.e., by kicking off all vCPUs and retrying, which should succeed on the second attempt.
TDX_INTERRUPTED_RESTARTABLE occurs when there is a pending interrupt on the host side during SEAMCALL tdh_mem_page_demote(). The approach is to retry indefinitely in KVM for TDX_INTERRUPTED_RESTARTABLE, because the interrupts are for host only in current exclusive kvm->mmu_lock path.
Page merging (page promotion)
-----------------------------
The RFC disallows the page merging on the mirror page table.
Unlike normal VMs, private memory in TDX requires the guest's ACCEPT operation. Therefore, transitioning from a non-leaf entry to a huge leaf entry in the S-EPT requires the non-leaf entry to be initially populated with small child entries, all in PENDING or ACCEPTED status. Subsequently, the merged huge leaf can be set to either PENDING or ACCEPTED status.
Therefore, counter-intuitively, converting a partial range (e.g., one 4KB) of a 2MB range from private to shared and then converting back to private does not result in a successful page promotion in the S-EPT. After converting a shared 4KB page back to private:
a) Linux Guest: Accepts the 4K page prior to accessing memory, prompting KVM to map it at the 4KB level, which prevents further EPT violations and avoids triggering page promotion.
b) Non-Linux Guest: May access the page before executing the ACCEPT operation. KVM identifies the physical page is 2MB contiguous and maps it at 2MB, causing a non-leaf to leaf transition in the mirror page table. However, after the preparation step, only 511 child entries in the S-EPT are in ACCEPTED status, with 1 newly mapped entry in PENDING status. The promotion request to the S-EPT fails due to this mixed status. If KVM re-enters the guest and triggers #VE for the guest to accept the page, the guest must accept the page at the 4KB level, as no 2MB mapping is available. After the ACCEPT operation, no further EPT violations occur to trigger page promotion.
So, also to avoid the comprehensive BUSY handling and rolling back code due to shared kvm->mmu_lock, the RFC disallows the page merging on the mirror page table. This should have minimal performance impact in practice, as up to now no page merging is observed for a real guest, except for the selftests.
Patches layout
==============
Patch 01: Glue code to [2].
It allows kvm_gmem_populate() and kvm_gmem_get_pfn() to get a 2MB private huge page from guest_memfd whenever GFN/index alignment, remaining size, and page attribute layout. Though this patch may not be needed after guest_memfd supporting in-place conversion in future, the guest_memfd needs to ensure something similar.
Patches 02-03: SEAMCALL changes under x86/virt.
Patches 04-09: Basic private huge page mapping/unmapping.
04: For build time, no huge pages, forced to 4KB.
05-07: Enhancements of tdx_clear_page(),tdx_reclaim_page,
tdx_wbinvd_page() to handle huge pages.
08: inc/dec folio ref count for huge pages.
The increasing of private folio ref count should be dropped after guest_memfd supporting in-place conversion. TDX will then only acquire private folio ref count upon errors during the page removing/reclaiming stage.
09: Turn on mapping/unmapping of huge pages for TD runtime.
Patch 10: Disallow page merging in the mirror page table. Patches 11-12: Allow guest's ACCEPT level to determine page mapping size. Patches 13-19: Basic page splitting support (with exclusive kvm->mmu_lock)
13: Enhance tdp_mmu_alloc_sp_split() for external page table 14: Add code to propagate splitting request to external page
table in tdp_mmu_set_spte(), which updates SPTE under exclusive kvm->mmu_lock.
15: TDX's counter part to patch 14. Implementation of hook
split_external_spt.
16-19: Split private huge pages for private-to-shared conversion
and punch hole.
Patches 20-21: Ignore page splitting request with shared kvm->mmu_lock
Main changes to [0]:
===================
- Disallow huge mappings in TD build time.
- Use hook private_max_mapping_level to convey TDX's mapping level info instead of having KVM MMU core to check certain bits in error_code to determine a fault's max_level.
- Move tdh_mem_range_block() for page splitting to TDX's implementation of hook split_external_spt.
- Do page splitting before tdp_mmu_zap_leafs(). So instead of BUG_ON() the tdp_mmu_zap_leafs(), out-of-memory failure for splitting can fail the ioctl KVM_SET_MEMORY_ATTRIBUTES or punch hole.
- Restrict page splitting to be under exclusive kvm->mmu_lock and ignore the page splitting under shared kvm->mmu_lock.
- Drop page merging support.
Testing
-------
The series is based on kvm/next.
This patchset is also available at: [3]
It is able to launch TDs with page demotion working correctly. Though it's still unable to trigger page promotion with a linux guest yet, the part of page promotion code is tested working with a selftest.
It's able to check huge mapping count in KVM at runtime at /sys/kernel/debug/kvm/pages_2m.
(Though this node includes huge mapping count for both shared and private memory, currently there're not many shared huge pages. In future, guest_memfd in-place conversion requires all shared pages to be 4KB. So there's no need to expand this interface).
[0] https://lore.kernel.org/all/[email protected]
[1] https://lore.kernel.org/lkml/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://github.com/intel/tdx/tree/huge_page_kvm_next_2025_04_23
Edgecombe, Rick P (1):
KVM: x86/mmu: Disallow page merging (huge page adjustment) for mirror
root
Isaku Yamahata (1):
KVM: x86/tdp_mmu: Alloc external_spt page for mirror page table
splitting
Xiaoyao Li (5):
x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
KVM: TDX: Enhance tdx_clear_page() to support huge pages
KVM: TDX: Assert the reclaimed pages were mapped as expected
KVM: TDX: Add a helper for WBINVD on huge pages with TD's keyID
KVM: TDX: Support huge page splitting with exclusive kvm->mmu_lock
Yan Zhao (14):
KVM: gmem: Allocate 2M huge page from guest_memfd backend x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
KVM: TDX: Enforce 4KB mapping level during TD build Time
KVM: TDX: Increase/decrease folio ref for huge pages
KVM: TDX: Enable 2MB mapping size after TD is RUNNABLE
KVM: x86: Add "vcpu" "gfn" parameters to x86 hook
private_max_mapping_level
KVM: TDX: Determine max mapping level according to vCPU's ACCEPT level
KVM: x86/tdp_mmu: Invoke split_external_spt hook with exclusive
mmu_lock
KVM: x86/mmu: Introduce kvm_split_boundary_leafs() to split boundary
leafs
KVM: Change the return type of gfn_handler_t() from bool to int
KVM: x86: Split huge boundary leafs before private to shared
conversion
KVM: gmem: Split huge boundary leafs for punch hole of private memory
KVM: x86: Force a prefetch fault's max mapping level to 4KB for TDX
KVM: x86: Ignore splitting huge pages in fault path for TDX
arch/arm64/kvm/mmu.c | 4 +- arch/loongarch/kvm/mmu.c | 4 +- arch/mips/kvm/mmu.c | 4 +- arch/powerpc/kvm/book3s.c | 4 +- arch/powerpc/kvm/e500_mmu_host.c | 4 +- arch/riscv/kvm/mmu.c | 4 +- arch/x86/include/asm/kvm-x86-ops.h | 1 + arch/x86/include/asm/kvm_host.h | 7 +- arch/x86/include/asm/tdx.h | 2 + arch/x86/kvm/mmu/mmu.c | 67 +++++--- arch/x86/kvm/mmu/mmu_internal.h | 2 +- arch/x86/kvm/mmu/paging_tmpl.h | 2 +- arch/x86/kvm/mmu/tdp_mmu.c | 200 +++++++++++++++++++---- arch/x86/kvm/mmu/tdp_mmu.h | 1 + arch/x86/kvm/svm/sev.c | 5 +- arch/x86/kvm/svm/svm.h | 5 +- arch/x86/kvm/vmx/main.c | 8 +- arch/x86/kvm/vmx/tdx.c | 244 +++++++++++++++++++++++------ arch/x86/kvm/vmx/tdx.h | 4 + arch/x86/kvm/vmx/tdx_arch.h | 3 + arch/x86/kvm/vmx/tdx_errno.h | 1 + arch/x86/kvm/vmx/x86_ops.h | 14 +- arch/x86/virt/vmx/tdx/tdx.c | 31 +++- arch/x86/virt/vmx/tdx/tdx.h | 1 + include/linux/kvm_host.h | 13 +- virt/kvm/guest_memfd.c | 183 ++++++++++------------ virt/kvm/kvm_main.c | 38 +++-- 27 files changed, 612 insertions(+), 244 deletions(-)
Signature · 1 line
2.43.2
RFC PATCH 1/21 KVM: gmem: Allocate 2M huge page from guest_memfd backend
Allocate 2M huge pages from the guest_memfd's filemap when the max_order is greater than or equal to PMD_ORDER.
Introduce a helper function, kvm_gmem_get_max_order(), to assist kvm_gmem_populate() and kvm_gmem_get_pfn() in obtaining a max_order, based on the alignment of GFN/index, range size and the consistency of page attributes.
Pass in the max_order to __kvm_gmem_get_pfn(), which invokes kvm_gmem_get_folio() to allocate a 2M huge page from gmem filemap if the max_order is >= PMD_ORDER. __kvm_gmem_get_pfn() then updates the max_order if the order of the allocated page is smaller than the requestd order.
Note:!!
This patch just serves as an glue layer on top of Michael Roth's series[1], showing TDX's basic assumptions to the guest_memfd, i.e.. guest_memfd allocates private huge pages whenever alignment of GFN/index, range size and the consistency of page attributes allow it.
As Dave mentioned at [2],
"Probably a good idea to focus on the long-term use case where we have in-place conversion support, and only allow truncation in hugepage (e.g., 2 MiB) size; conversion shared<->private could still be done on 4 KiB granularity as for hugetlb.",
"In general, I think our time is better spent
working on the real deal than on interim solutions that should not be called "THP support",
Please don't spend much time on reviewing this patch, as it would propably be gone or appear in another form after guest_memfd's solution based on hugetlb for in-place-conversion is available.
--- virt/kvm/guest_memfd.c | 153 +++++++++++++++-------------------------- 1 file changed, 56 insertions(+), 97 deletions(-)
virt/kvm/guest_memfd.c +56 -97
@@ -265,36 +265,6 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct file *file,return r; } -static struct folio *kvm_gmem_get_huge_folio (struct inode *inode,pgoff_t index, -unsigned int order) -{ -pgoff_t npages =1UL << order; -pgoff_t huge_index =round_down (index, npages); -struct address_space *mapping = inode->i_mapping; -gfp_t gfp =mapping_gfp_mask (mapping) | __GFP_NOWARN; -loff_t size =i_size_read (inode); -struct folio *folio; - -/* Make sure hugepages would be fully-contained by inode */ -if ((huge_index + npages) *PAGE_SIZE > size) -return NULL ; - -if (filemap_range_has_page (mapping, (loff_t )huge_index <<PAGE_SHIFT , - (loff_t)(huge_index + npages -1 ) <<PAGE_SHIFT )) -return NULL ; - - folio =filemap_alloc_folio (gfp, order); -if (!folio) -return NULL ; - -if (filemap_add_folio (mapping, folio, huge_index, gfp)) { -folio_put (folio); -return NULL ; - } - -return folio; -} -/* * Returns a locked folio on success. The caller is responsible for * setting the up-to-date flag before the memory is mapped into the guest.
@@ -304,14 +274,19 @@ static struct folio *kvm_gmem_get_huge_folio(struct inode *inode, pgoff_t index,* Ignore accessed, referenced, and dirty flags. The memory is * unevictable and there is no storage to write back to. */ -static struct folio *kvm_gmem_get_folio (struct inode *inode,pgoff_t index) +static struct folio *kvm_gmem_get_folio (struct inode *inode,pgoff_t index,int max_order) {struct folio *folio =NULL ; -if (gmem_2m_enabled) - folio =kvm_gmem_get_huge_folio (inode, index,PMD_ORDER ); +if (max_order >=PMD_ORDER ) { +fgf_t fgp_flags =FGP_LOCK |FGP_ACCESSED |FGP_CREAT ; -if (!folio) + fgp_flags |=fgf_set_order (1U << (PAGE_SHIFT +PMD_ORDER )); + folio =__filemap_get_folio (inode->i_mapping, index, fgp_flags, +mapping_gfp_mask (inode->i_mapping)); + } + +if (!folio ||IS_ERR (folio)) folio =filemap_grab_folio (inode->i_mapping, index);return folio;
@@ -402,49 +377,11 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)static long kvm_gmem_allocate (struct inode *inode,loff_t offset,loff_t len) { -struct address_space *mapping = inode->i_mapping; -pgoff_t start, index, end; -int r; - -/* Dedicated guest is immutable by default. */ -if (offset + len >i_size_read (inode)) -return -EINVAL ; - -filemap_invalidate_lock_shared (mapping); - - start = offset >>PAGE_SHIFT ; - end = (offset + len) >>PAGE_SHIFT ; - - r =0 ; -for (index = start; index < end; ) { -struct folio *folio; - -if (signal_pending (current)) { - r = -EINTR ; -break ; - } - - folio =kvm_gmem_get_folio (inode, index); -if (IS_ERR (folio)) { - r =PTR_ERR (folio); -break ; - } - - index =folio_next_index (folio); - -folio_unlock (folio); -folio_put (folio); - -/* 64-bit only, wrapping the index should be impossible. */ -if (WARN_ON_ONCE (!index)) -break ; - -cond_resched (); - } - -filemap_invalidate_unlock_shared (mapping); - -return r; +/* +* Skip supporting allocate for now. This can be added easiler after +* __kvm_gmem_get_pfn() is settled down. +*/ +return -EOPNOTSUPP ; } static long kvm_gmem_fallocate(struct file *file, int mode, loff_t offset,
@@ -853,7 +790,7 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file, huge_index + (1ull << *max_order) > slot->gmem.pgoff + slot->npages) *max_order =0 ; - folio =kvm_gmem_get_folio (file_inode (file), index); + folio =kvm_gmem_get_folio (file_inode (file), index, *max_order);if (IS_ERR (folio))return folio;
@@ -869,6 +806,40 @@ static struct folio *__kvm_gmem_get_pfn(struct file *file,return folio; } +static int kvm_gmem_get_max_order (struct kvm *kvm,struct kvm_memory_slot *slot, +gfn_t gfn,pgoff_t index,long npages,int *max_order) +{ +int ret =0 ; +int order =0 ; +/* +* The max order shouldn't extend beyond the GFN range being +* populated in this iteration, so set max_order accordingly. +* __kvm_gmem_get_pfn() will then further adjust the order to +* one that is contained by the backing memslot/folio. +*/ + order =0 ; + +while (IS_ALIGNED (gfn,1 << (order +1 )) && (npages >= (1 << (order +1 )))) + order++; + + order =min (order,PMD_ORDER ); + +while (!kvm_range_has_memory_attributes (kvm, gfn, gfn + (1 << order), +KVM_MEMORY_ATTRIBUTE_PRIVATE , +KVM_MEMORY_ATTRIBUTE_PRIVATE )) { +if (!order) { + ret = -ENOENT ; +return ret; + } + order--; + } + +WARN_ON (!IS_ALIGNED (index,1 << order) || (npages < (1 << order))); + + *max_order = order; +return ret; +} +int kvm_gmem_get_pfn (struct kvm *kvm,struct kvm_memory_slot *slot,gfn_t gfn,kvm_pfn_t *pfn,struct page **page,int *max_order)
@@ -882,15 +853,8 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,if (!file)return -EFAULT ; -/* -* The caller might pass a NULL 'max_order', but internally this -* function needs to be aware of any order limitations set by -* __kvm_gmem_get_pfn() so the scope of preparation operations can -* be limited to the corresponding range. The initial order can be -* arbitrarily large, but gmem doesn't currently support anything -* greater than PMD_ORDER so use that for now. -*/ - max_order_local =PMD_ORDER ; +kvm_gmem_get_max_order (kvm, slot, gfn, index, slot->npages - (gfn - slot->base_gfn), + &max_order_local); folio =__kvm_gmem_get_pfn (file, slot, index, pfn, &max_order_local);if (IS_ERR (folio)) {
@@ -953,6 +917,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, longbreak ; } + ret =kvm_gmem_get_max_order (kvm, slot, gfn,kvm_gmem_get_index (slot, gfn), + npages - i, &max_order); +if (ret) +break ; + folio =__kvm_gmem_get_pfn (file, slot, index, &pfn, &max_order);if (IS_ERR (folio)) { ret =PTR_ERR (folio);
@@ -967,17 +936,8 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long }folio_unlock (folio); -WARN_ON (!IS_ALIGNED (gfn,1 << max_order) || - (npages - i) < (1 << max_order)); ret = -EINVAL ; -while (!kvm_range_has_memory_attributes (kvm, gfn, gfn + (1 << max_order), -KVM_MEMORY_ATTRIBUTE_PRIVATE , -KVM_MEMORY_ATTRIBUTE_PRIVATE )) { -if (!max_order) - goto put_folio_and_exit; - max_order--; - } p = src ? src + i *PAGE_SIZE :NULL ; ret =post_populate (kvm, gfn, pfn, p, max_order, opaque);
@@ -986,7 +946,6 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, longkvm_gmem_mark_prepared (file, index, max_order); } -put_folio_and_exit:folio_put (folio);if (ret)break ;
Signature · 1 line
2.43.2
RFC PATCH 2/21 x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
In reply to Yan ZhaoEnhance the SEAMCALL wrapper tdh_mem_page_aug() to support huge pages.
Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.
--- arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/x86/virt/vmx/tdx/tdx.c +10 -1
@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page); +unsigned long idx =0 ;u64 ret; -tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx (folio, page) + nr_pages >folio_nr_pages (folio))) +return -EINVAL ; + +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++)); + ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;
Signature · 1 line
2.43.2
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 21 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +0
-0
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1598
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 );
PTE_SHIFT.
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1603
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-1+struct folio *folio =page_folio (page); +unsigned long idx =0 ;u64 ret; -tdx_clflush_page (page); + if (!(level >= TDX_PS_4K && level < TDX_PS_NR) ||
Do we even need this check?
Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +6
-0+ (folio_page_idx(folio, page) + nr_pages > folio_nr_pages(folio))) +return -EINVAL ; + +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++)); + ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;Signature · 1 line
2.43.2
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 17 lines · Kirill A. Shutemov
Quoted 16 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
Quoted 11 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1598
Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx.c +1
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 );
PTE_SHIFT.
Yes. Thanks.
Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1603
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-1+struct folio *folio =page_folio (page); +unsigned long idx =0 ;u64 ret; -tdx_clflush_page (page); + if (!(level >= TDX_PS_4K && level < TDX_PS_NR) ||Do we even need this check?
Maybe not if tdh_mem_page_aug() trusts KVM :)
The consideration is to avoid nr_pages being too huge to cause too many tdx_clflush_page()s on any reckless error.
Quoted 15 lines
Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +6
-0+ (folio_page_idx(folio, page) + nr_pages > folio_nr_pages(folio))) +return -EINVAL ; + +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++)); + ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;Signature · 1 line
2.43.2Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
In reply to Yan ZhaoQuoted 26 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +2
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page);
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1608
Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +7
-1+unsigned long idx =0 ;u64 ret; -tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx (folio, page) + nr_pages >folio_nr_pages (folio))) +return -EINVAL ; + +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++));
Is the following better to save a variable?
while (nr_pages)
tdx_clflush_page(nth_page(page, --nr_pages));
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1
-0+ ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 40 lines · Binbin Wu
Quoted 25 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +1
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 );arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1607
Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +8
-1+struct folio *folio =page_folio (page); +unsigned long idx =0 ;u64 ret; -tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx (folio, page) + nr_pages >folio_nr_pages (folio))) +return -EINVAL ; + +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++));Is the following better to save a variable?
Quoted 2 lines
while (
nr_pages)
tdx_clflush_page(nth_page(page, --nr_pages));
Looks better except performing the clflush in reverse order :)
Quoted 5 lines
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1
-0+ ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
In reply to Yan ZhaoQuoted 8 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.
This should have a brief background on why it doesn't use the arg - what is deficient today. Also, an explanation of how it will be used (i.e. what types of pages will be passed)
Quoted 21 lines
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>---
arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++-1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +3
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page); +unsigned long idx =0 ; u64 ret;
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1605
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-1-tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx(folio, page) + nr_pages > folio_nr_pages(folio))) +return -EINVAL ;
Shouldn't KVM not try to map a huge page in this situation? Doesn't seem like a job for the SEAMCALL wrapper.
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1608
Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-0+ +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++));
clflush_cache_range() is:
static void tdx_clflush_page(struct page *page)
{
clflush_cache_range(page_to_virt(page), PAGE_SIZE);
}So we have loops within loops... Better to add an arg to tdx_clflush_page() or add a variant that takes one.
Quoted 4 lines
+
ret =seamcall_ret(TDH_MEM_PAGE_AUG, &args);
*ext_err1= args.rcx;
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 6 lines · Rick P
Quoted 5 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
Quoted 7 lines
Quoted 3 lines
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.This should have a brief background on why it doesn't use the arg - what is deficient today. Also, an explanation of how it will be used (i.e. what types of pages will be passed)
Will do.
Quoted 20 lines
Quoted 20 lines
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>---
arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++-1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +3
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page); +unsigned long idx =0 ; u64 ret;
Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1605
Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-1-tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx(folio, page) + nr_pages > folio_nr_pages(folio))) +return -EINVAL ;Shouldn't KVM not try to map a huge page in this situation? Doesn't seem like a job for the SEAMCALL wrapper.
Ok. If the decision is to trust KVM and all potential callers, it's reasonable to drop those checks.
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1608
Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-0+ +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++));
clflush_cache_range()is:
static voidtdx_clflush_page(struct page *page)
Quoted 6 lines
{ clflush_cache_range(page_to_virt(page), PAGE_SIZE); }So we have loops within loops... Better to add an arg to
tdx_clflush_page()or add a variant that takes one.
Ok.
One thing to note is that even with an extra arg, tdx_clflush_page() has to call clflush_cache_range() page by page because with
"#if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)", page virtual addresses are not necessarily contiguous.
What about Binbin's proposal [1]? i.e.,
while (nr_pages)
tdx_clflush_page(nth_page(page, --nr_pages));
[1] https://lore.kernel.org/all/[email protected]/
Quoted 5 lines
Quoted 4 lines
+
ret =seamcall_ret(TDH_MEM_PAGE_AUG, &args);
*ext_err1= args.rcx;
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 11 lines · Yan Zhao
Quoted 3 lines
So we have loops within loops... Better to add an arg to
tdx_clflush_page()or
add a variant that takes one.Ok.
One thing to note is that even with an extra arg,
tdx_clflush_page()has to callclflush_cache_range()page by page because with
"#ifdefined(CONFIG_SPARSEMEM)&& !defined(CONFIG_SPARSEMEM_VMEMMAP)", page virtual addresses are not necessarily contiguous.
Quoted 7 lines
What about Binbin's proposal [1]? i.e.,
while (
nr_pages)tdx_clflush_page(nth_page(page, --nr_pages));
These SEAMCALLs are handling physically contiguous pages so I don't think we need to worry about that. But Binbin's suggestion seems fine too.
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
In reply to Yan ZhaoQuoted 66 lines · Yan Zhao
Quoted 6 lines · Rick P
Quoted 5 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
Quoted 7 lines
Quoted 3 lines
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.This should have a brief background on why it doesn't use the arg - what is deficient today. Also, an explanation of how it will be used (i.e. what types of pages will be passed)
Will do.
Quoted 21 lines
Quoted 21 lines
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/virt/vmx/tdx/tdx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +3
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page); +unsigned long idx =0 ; u64 ret;Quoted 7 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1605
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-1-tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx(folio, page) + nr_pages > folio_nr_pages(folio))) +return -EINVAL ;Shouldn't KVM not try to map a huge page in this situation? Doesn't seem like a job for the SEAMCALL wrapper.
Ok. If the decision is to trust KVM and all potential callers, it's reasonable to drop those checks.
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1608
Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-0+ +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++));
clflush_cache_range()is:
static voidtdx_clflush_page(struct page *page)Quoted 6 lines
{ clflush_cache_range(page_to_virt(page), PAGE_SIZE); }So we have loops within loops... Better to add an arg to
tdx_clflush_page()or add a variant that takes one.Ok.
One thing to note is that even with an extra arg,
tdx_clflush_page()has to callclflush_cache_range()page by page because with
"#ifdefined(CONFIG_SPARSEMEM)&& !defined(CONFIG_SPARSEMEM_VMEMMAP)", page virtual addresses are not necessarily contiguous.
Quoted 4 lines
What about Binbin's proposal [1]? i.e.,
while (
nr_pages)tdx_clflush_page(nth_page(page, --nr_pages));
What's the problem with using:
+ for (int i = 0; nr_pages; nr_pages--)
+ tdx_clflush_page(nth_page(page, i++))
The kernel now allows C99-style definition of variables inside a loop + it's clear how many times the loop has to be executed.
Quoted 9 lines
[1] https://lore.kernel.org/all/[email protected]/
Quoted 5 lines
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1
-0+ ret = seamcall_ret(TDH_MEM_PAGE_AUG, &args); *ext_err1 = args.rcx;
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Review
Quoted 9 lines · Nikolay Borisov
Quoted 4 lines
What about Binbin's proposal [1]? i.e.,
while (
nr_pages)tdx_clflush_page(nth_page(page, --nr_pages));What's the problem with using:
(quoted diff) +2
-0+ for (int i = 0; nr_pages; nr_pages--) + tdx_clflush_page(nth_page(page, i++))
Thanks! It looks good to me.
Quoted 2 lines
The kernel now allows C99-style definition of variables inside a loop + it's clear how many times the loop has to be executed.
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
In reply to Yan ZhaoQuoted 23 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>---
arch/x86/virt/vmx/tdx/tdx.c| 11 ++++++++++-1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +0
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page),
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1605
Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +6
-1}; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page); +unsigned long idx =0 ;u64 ret; -tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx (folio, page) + nr_pages >folio_nr_pages (folio))) +return -EINVAL ;
Returning -EINVAL looks incorrect as the return type is u64.
Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +4
-0+ +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++)); + ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;Signature · 1 line
2.43.2
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 30 lines · Chao Gao
Quoted 29 lines · Yan Zhao
Enhance the SEAMCALL wrapper
tdh_mem_page_aug()to support huge pages.Verify the validity of the level and ensure that the mapping range is fully contained within the page folio.
As a conservative solution, perform CLFLUSH on all pages to be mapped into the TD before invoking the SEAMCALL
TDH_MEM_PAGE_AUG. This ensures that any dirty cache lines do not write back later and clobber TD memory.Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>---
arch/x86/virt/vmx/tdx/tdx.c| 11 ++++++++++-1 file changed, 10 insertions(+), 1 deletion(-)arch/x86/virt/vmx/tdx/tdx.c +3
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u .rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page); +unsigned long idx =0 ;u64 ret;
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u · line 1605
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-1-tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx (folio, page) + nr_pages >folio_nr_pages (folio))) +return -EINVAL ;Returning
-EINVALlooks incorrect as the return type is u64.
Good catch. Thanks!
I'll think about how to handle it. Looks it could be dropped if we trust KVM.
Quoted 9 lines
Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx.c +3
-0+while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++)); + ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;Signature · 1 line
2.43.2
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
In reply to Yan Zhaoarch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u
Quoted 5 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +0
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u
According to the discussion in DPAMT [*],
"hpa here points to a 2M region that pamt_pages covers. We don't have struct page that represents it. Passing 4k struct page would be misleading IMO."
Should we update tdh_mem_page_aug() accordingly to use hpa? Or use struct folio instead?
[*] https://lore.kernel.org/all/3coaqkcfp7xtpvh2x4kph55qlopupknm7dmzqox6fakzaedhem@a2oysbvbshpm/
Quoted 22 lines
arch/x86/virt/vmx/tdx/tdx.c +10
-1.rdx = tdx_tdr_pa(td), .r8 = page_to_phys(page), }; +unsigned long nr_pages =1 << (level *9 ); +struct folio *folio =page_folio (page); +unsigned long idx =0 ;u64 ret; -tdx_clflush_page (page); +if (!(level >=TDX_PS_4K && level <TDX_PS_NR ) || + (folio_page_idx (folio, page) + nr_pages >folio_nr_pages (folio))) +return -EINVAL ; + +while (nr_pages--) +tdx_clflush_page (nth_page (page, idx++)); + ret =seamcall_ret (TDH_MEM_PAGE_AUG , &args); *ext_err1 = args.rcx;Signature · 1 line
2.43.2
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 11 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u
Quoted 5 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +0
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, uAccording to the discussion in DPAMT [*],
"hpa here points to a 2M region thatpamt_pagescovers. We don't have struct page that represents it. Passing 4k struct page would be misleading IMO."
Quoted 4 lines
Should we update
tdh_mem_page_aug()accordingly to use hpa? Or use struct folio instead?[*] https://lore.kernel.org/all/3coaqkcfp7xtpvh2x4kph55qlopupknm7dmzqox6fakzaedhem@a2oysbvbshpm/
The original seamcall wrapper patches used "u64 hpa", etc everywhere. The feedback was that it was too error prone to not have types. We looked at using kvm types (hpa_t, etc), but the type checking was still just surface level [0].
So the goal is to reduce errors and improve code readability. We can consider breaking symmetry if it is better that way. In this case though, why not use struct folio?
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 18 lines · Rick P
Quoted 11 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u
Quoted 5 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +0
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, uAccording to the discussion in DPAMT [*],
"hpa here points to a 2M region thatpamt_pagescovers. We don't have struct page that represents it. Passing 4k struct page would be misleading IMO."Quoted 4 lines
Should we update
tdh_mem_page_aug()accordingly to use hpa? Or use struct folio instead?[*] https://lore.kernel.org/all/3coaqkcfp7xtpvh2x4kph55qlopupknm7dmzqox6fakzaedhem@a2oysbvbshpm/
Quoted 7 lines
The original seamcall wrapper patches used "u64 hpa", etc everywhere. The feedback was that it was too error prone to not have types. We looked at using kvm types (
hpa_t, etc), but the type checking was still just surface level [0].So the goal is to reduce errors and improve code readability. We can consider breaking symmetry if it is better that way. In this case though, why not use struct folio?
My vote would be to prefer using "hpa" and not rely on folio/page structs for guest_memfd allocated memory wherever possible.
Quoted 2 lines
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 6 lines · Vishal Annapurve
Quoted 6 lines
The original seamcall wrapper patches used "u64 hpa", etc everywhere. The feedback was that it was too error prone to not have types. We looked at using
kvm types (hpa_t, etc), but the type checking was still just surface level [0].
Quoted 7 lines
Quoted 4 lines
So the goal is to reduce errors and improve code readability. We can consider
breaking symmetry if it is better that way. In this case though, why not use struct folio?My vote would be to prefer using "hpa" and not rely on folio/page structs for
guest_memfdallocated memory wherever possible.
Is this because you want to enable struct page-less gmemfd in the future? Or other reason?
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 13 lines · Rick P
Quoted 11 lines · Vishal Annapurve
Quoted 10 lines
The original seamcall wrapper patches used "u64 hpa", etc everywhere. The feedback was that it was too error prone to not have types. We looked at using
kvm types (hpa_t, etc), but the type checking was still just surface level [0].So the goal is to reduce errors and improve code readability. We can consider
breaking symmetry if it is better that way. In this case though, why not use struct folio?
Quoted 4 lines
Quoted 2 lines
My vote would be to prefer using "hpa" and not rely on folio/page structs for
guest_memfdallocated memory wherever possible.Is this because you want to enable struct page-less gmemfd in the future?
Yes. That's the only reason.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 6 lines · Vishal Annapurve
Quoted 4 lines
Quoted 2 lines
My vote would be to prefer using "hpa" and not rely on folio/page structs for
guest_memfdallocated memory wherever possible.Is this because you want to enable struct page-less gmemfd in the future?
Yes. That's the only reason.
I don't think we should change just this field of this seamcall wrapper from the current pattern for that reason. When this stuff comes along it will be just about as easy to change it with the rest. Then in the meantime it doesn't look asymmetric.
In general, I (again) think that we should not focus on accommodating future stuff unless there is an ABI touch point. This is to ultimately speed enabling of the entire stack.
It is definitely not to make it harder to implement TDX support for pfn based gmem in the future. Rather to make it possible. As in, if nothing is upstream because we are endlessly debating how it all fits together at once, then it won't be possible to enhance it further.
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 14 lines · Rick P
Quoted 6 lines · Vishal Annapurve
Quoted 4 lines
Quoted 2 lines
My vote would be to prefer using "hpa" and not rely on folio/page structs for
guest_memfdallocated memory wherever possible.Is this because you want to enable struct page-less gmemfd in the future?
Yes. That's the only reason.
I don't think we should change just this field of this seamcall wrapper from the current pattern for that reason. When this stuff comes along it will be just about as easy to change it with the rest. Then in the meantime it doesn't look asymmetric.
Quoted 8 lines
In general, I (again) think that we should not focus on accommodating future stuff unless there is an ABI touch point. This is to ultimately speed enabling of the entire stack.
It is definitely not to make it harder to implement TDX support for pfn based gmem in the future. Rather to make it possible. As in, if nothing is upstream because we are endlessly debating how it all fits together at once, then it won't be possible to enhance it further.
I agree and if we can't do without page struct for now that's fine. My response was just to favor pfn/hpa over "page struct" if possible, given that we have a choice here. Feel free to ignore if symmetry seems more important.
Re: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
In reply to Edgecombe, Rick PQuoted 17 lines · Rick P
Quoted 11 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u
Quoted 5 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +0
-0@@ -1595,9 +1595,18 @@ u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, uAccording to the discussion in DPAMT [*],
"hpa here points to a 2M region thatpamt_pagescovers. We don't have struct page that represents it. Passing 4k struct page would be misleading IMO."Quoted 4 lines
Should we update
tdh_mem_page_aug()accordingly to use hpa? Or use struct folio instead?[*] https://lore.kernel.org/all/3coaqkcfp7xtpvh2x4kph55qlopupknm7dmzqox6fakzaedhem@a2oysbvbshpm/
Quoted 7 lines
The original seamcall wrapper patches used "u64 hpa", etc everywhere. The feedback was that it was too error prone to not have types. We looked at using kvm types (
hpa_t, etc), but the type checking was still just surface level [0].So the goal is to reduce errors and improve code readability. We can consider breaking symmetry if it is better that way. In this case though, why not use struct folio?
I'm Ok with using struct folio.
My previous ask was based on 2 considerations:
1. hpa is simpler and I didn't find Dave's NAK to Kirill's patch (v1 or v2).
2. using struct folio, I need to introduce "start_idx" as well (as below), because it's likely that guest_memfd provides a huge folio while KVM wants to map it at 4KB.
u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct folio *folio,
unsigned long start_idx, u64 *ext_err1, u64 *ext_err2)
{
struct page *start = folio_page(folio, start_idx);
unsigned long npages = 1 << (level * PTE_SHIFT);
struct tdx_module_args args = {
.rcx = gpa | level,
.rdx = tdx_tdr_pa(td),
.r8 = page_to_phys(start),
};
u64 ret;
if (start_idx + npages > folio_nr_pages(folio))
return TDX_SW_ERROR;
for (int i = 0; i < npages; i++)
tdx_clflush_page(nth_page(start, i));
ret = seamcall_ret(TDH_MEM_PAGE_AUG, &args);
*ext_err1 = args.rcx;
*ext_err2 = args.rdx;
return ret;
} Quoted 1 line
Edgecombe, Rick P
maintainerRe: [RFC PATCH 02/21] x86/virt/tdx: Enhance tdh_mem_page_aug() to support huge pages
Quoted 3 lines · Yan Zhao
2. using struct folio, I need to introduce "
start_idx" as well (as below), because it's likely thatguest_memfdprovides a huge folio while KVM wants to map it at 4KB.
Seems ok to me.
Quoted 26 lines
u64 tdh_mem_page_aug(struct
tdx_td*td, u64 gpa, int level, struct folio *folio, unsigned longstart_idx, u64*ext_err1, u64*ext_err2) {
struct page *start =folio_page(folio, start_idx);
unsigned long npages = 1 << (level *PTE_SHIFT);
structtdx_module_argsargs = {
.rcx = gpa | level,
.rdx =tdx_tdr_pa(td),
.r8 =page_to_phys(start),
};
u64 ret;
if (start_idx+ npages >folio_nr_pages(folio))
returnTDX_SW_ERROR;
for (int i = 0; i < npages; i++)tdx_clflush_page(nth_page(start, i));
ret =seamcall_ret(TDH_MEM_PAGE_AUG, &args);
*ext_err1= args.rcx;
*ext_err2= args.rdx;
return ret;
}
RFC PATCH 3/21 x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
In reply to Yan ZhaoFrom: Xiaoyao Li <[email protected]>
Add a wrapper tdh_mem_page_demote() to invoke SEAMCALL TDH_MEM_PAGE_DEMOTE to demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M guest private pages.
The "gpa" and "level" direct the TDX module to search and find the old huge leaf entry.
As the new non-leaf entry points to a page table page, callers need to pass in the page table page in parameter "page".
In case of S-EPT walk failure, the entry, level and state where the error was detected are returned in ext_err1 and ext_err2.
On interrupt pending, SEAMCALL TDH_MEM_PAGE_DEMOTE returns error TDX_INTERRUPTED_RESTARTABLE.
[Yan: Rebased and split patch, wrote changelog]
--- arch/x86/include/asm/tdx.h | 2 ++ arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++ arch/x86/virt/vmx/tdx/tdx.h | 1 + 3 files changed, 23 insertions(+)
arch/x86/include/asm/tdx.h +2 -0
@@ -177,6 +177,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);u64 tdh_mng_create (struct tdx_td *td,u16 hkid);u64 tdh_vp_create (struct tdx_td *td,struct tdx_vp *vp);u64 tdh_mng_rd (struct tdx_td *td,u64 field,u64 *data); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_extend (struct tdx_td *td,u64 gpa,u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_finalize (struct tdx_td *td);u64 tdh_vp_flush (struct tdx_vp *vp);
arch/x86/virt/vmx/tdx/tdx.c +20 -0
@@ -1684,6 +1684,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) }EXPORT_SYMBOL_GPL (tdh_mng_rd); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2) +{ +struct tdx_module_args args = { + .rcx = gpa | level, + .rdx =tdx_tdr_pa (td), + .r8 =page_to_phys (page), + }; +u64 ret; + +tdx_clflush_page (page); + ret =seamcall_ret (TDH_MEM_PAGE_DEMOTE , &args); + + *ext_err1 = args.rcx; + *ext_err2 = args.rdx; + +return ret; +} +EXPORT_SYMBOL_GPL (tdh_mem_page_demote); +u64 tdh_mr_extend (struct tdx_td *td,u64 gpa,u64 *ext_err1,u64 *ext_err2) {struct tdx_module_args args = {
arch/x86/virt/vmx/tdx/tdx.h +1 -0
@@ -24,6 +24,7 @@#define TDH_MNG_KEY_CONFIG 8#define TDH_MNG_CREATE 9#define TDH_MNG_RD 11 +#define TDH_MEM_PAGE_DEMOTE 15#define TDH_MR_EXTEND 16#define TDH_MR_FINALIZE 17#define TDH_VP_FLUSH 18
Signature · 1 line
2.43.2
Re: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
Quoted 8 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M
2M or 4K?
Quoted 1 line
guest private pages.
[...]
Re: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
Quoted 5 lines · Binbin Wu
Quoted 2 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Quoted 8 lines
Quoted 6 lines
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M2M or 4K?
The 512 leaf entries point to 2M guest private pages together, each pointing to 4K.
Quoted 1 line
Quoted 1 line
guest private pages.
Re: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
Quoted 5 lines · Yan Zhao
Quoted 4 lines · Binbin Wu
Quoted 2 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M2M or 4K?
The 512 leaf entries point to 2M guest private pages together,
If this, it should be 2M range, since it's not a huge page after demotion. Also, the plural "pages" is confusing.
Quoted 4 lines
each pointing to
4K.Quoted 1 line
Quoted 1 line
guest private pages.
Re: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
Quoted 14 lines · Binbin Wu
Quoted 11 lines · Yan Zhao
Quoted 10 lines · Binbin Wu
Quoted 8 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
2M or 4K?
The 512 leaf entries point to 2M guest private pages together,
If this, it should be 2M range, since it's not a huge page after demotion. Also, the plural "pages" is confusing.
Ah, indeed, plural "pages" is confiusing :)
Maybe below is better:
The newly linked page table now contains 512 leaf entries, each pointing to a 4K guest private page within the 2M range.
Quoted 5 lines
Quoted 4 lines
each pointing to
4K.Quoted 1 line
Quoted 1 line
guest private pages.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
In reply to Yan ZhaoQuoted 16 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M guest private pages.The "gpa" and "level" direct the TDX module to search and find the old huge leaf entry.
As the new non-leaf entry points to a page table page, callers need to pass in the page table page in parameter "page".
Quoted 7 lines
In case of S-EPT walk failure, the entry, level and state where the error was detected are returned in
ext_err1andext_err2.On interrupt pending, SEAMCALL
TDH_MEM_PAGE_DEMOTEreturns errorTDX_INTERRUPTED_RESTARTABLE.[Yan: Rebased and split patch, wrote changelog]
We should add the level of detail here like we did for the base series ones.
Quoted 28 lines
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/include/asm/tdx.h | 2 ++ arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++ arch/x86/virt/vmx/tdx/tdx.h | 1 + 3 files changed, 23 insertions(+)arch/x86/include/asm/tdx.h +2
-0@@ -177,6 +177,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);u64 tdh_mng_create (struct tdx_td *td,u16 hkid);u64 tdh_vp_create (struct tdx_td *td,struct tdx_vp *vp);u64 tdh_mng_rd (struct tdx_td *td,u64 field,u64 *data); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_extend (struct tdx_td *td,u64 gpa,u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_finalize (struct tdx_td *td);u64 tdh_vp_flush (struct tdx_vp *vp);arch/x86/virt/vmx/tdx/tdx.c +0
-0
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1691
Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +5
-0@@ -1684,6 +1684,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) }EXPORT_SYMBOL_GPL (tdh_mng_rd); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2) +{ +struct tdx_module_args args = { + .rcx = gpa | level,
This will only ever be level 2MB, how about dropping the arg?
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1701
Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10
-0+ .rdx = tdx_tdr_pa(td), + .r8 = page_to_phys(page), + }; +u64 ret; + +tdx_clflush_page (page); + ret =seamcall_ret (TDH_MEM_PAGE_DEMOTE , &args); + + *ext_err1 = args.rcx; + *ext_err2 = args.rdx;
How about we just call these entry and level_state, like the caller.
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1705
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (tdh_mem_page_demote);
Looking in the docs, TDX module gives some somewhat constrained guidance:
1. TDH.MEM.PAGE.DEMOTE should be invoked in a loop until it terminates successfully.
2. The host VMM should be designed to avoid cases where interrupt storms prevent successful completion of TDH.MEM.PAGE.DEMOTE.
The caller looks like:
do {
err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
&entry, &level_state);
} while (err == TDX_INTERRUPTED_RESTARTABLE);
if (unlikely(tdx_operand_busy(err))) {
tdx_no_vcpus_enter_start(kvm);
err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
&entry, &level_state);
tdx_no_vcpus_enter_stop(kvm);
}The brute force second case could also be subjected to a TDX_INTERRUPTED_RESTARTABLE and is not handled. As for interrupt storms, I guess we could disable interrupts while we do the second brute force case. So the TDX_INTERRUPTED_RESTARTABLE loop could have a max retries, and the brute force case could also disable interrupts.
Hmm, how to pick the max retries count. It's a tradeoff between interrupt latency and DOS/code complexity. Do we have any idea how long demote might take?
Re: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
Quoted 20 lines · Rick P
Quoted 19 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M guest private pages.The "gpa" and "level" direct the TDX module to search and find the old huge leaf entry.
As the new non-leaf entry points to a page table page, callers need to pass in the page table page in parameter "page".
In case of S-EPT walk failure, the entry, level and state where the error was detected are returned in
ext_err1andext_err2.
Quoted 6 lines
Quoted 4 lines
On interrupt pending, SEAMCALL
TDH_MEM_PAGE_DEMOTEreturns errorTDX_INTERRUPTED_RESTARTABLE.[Yan: Rebased and split patch, wrote changelog]
We should add the level of detail here like we did for the base series ones.
I'll provide changelog details under "---" of each patch in the next version.
Quoted 31 lines
Quoted 31 lines
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/include/asm/tdx.h | 2 ++ arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++ arch/x86/virt/vmx/tdx/tdx.h | 1 + 3 files changed, 23 insertions(+)arch/x86/include/asm/tdx.h +2
-0@@ -177,6 +177,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);u64 tdh_mng_create (struct tdx_td *td,u16 hkid);u64 tdh_vp_create (struct tdx_td *td,struct tdx_vp *vp);u64 tdh_mng_rd (struct tdx_td *td,u64 field,u64 *data); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_extend (struct tdx_td *td,u64 gpa,u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_finalize (struct tdx_td *td);u64 tdh_vp_flush (struct tdx_vp *vp);arch/x86/virt/vmx/tdx/tdx.c +0
-0@@ -1684,6 +1684,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) }EXPORT_SYMBOL_GPL (tdh_mng_rd);
Quoted 7 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1691
Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +5
-0+u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2) +{ +struct tdx_module_args args = { + .rcx = gpa | level,This will only ever be level 2MB, how about dropping the arg?
Do you mean hardcoding level to be 2MB in tdh_mem_page_demote()?
The SEAMCALL TDH_MEM_PAGE_DEMOTE supports level of 1GB in current TDX module.
Quoted 4 lines
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +4
-0+ .rdx = tdx_tdr_pa(td), + .r8 = page_to_phys(page), + }; +u64 ret;
Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1701
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +6
-0+ +tdx_clflush_page (page); + ret =seamcall_ret (TDH_MEM_PAGE_DEMOTE , &args); + + *ext_err1 = args.rcx; + *ext_err2 = args.rdx;How about we just call these entry and
level_state, like the caller.
Not sure, but I feel that ext_err* might be better, because
- according to the spec,
a) the args.rcx, args.rdx is unmodified (i.e. still hold the passed-in value) in case of error TDX_INTERRUPTED_RESTARTABLE.
b) args.rcx, args.rdx can only be interpreted as entry and level_state in case of EPT walk error.
c) in other cases, they are 0.
- consistent with tdh_mem_page_aug(), tdh_mem_range_block()...
Quoted 24 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1705
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (tdh_mem_page_demote);Looking in the docs, TDX module gives some somewhat constrained guidance:
1. TDH.MEM.PAGE.DEMOTE should be invoked in a loop until it terminates successfully.
2. The host VMM should be designed to avoid cases where interrupt storms prevent successful completion of TDH.MEM.PAGE.DEMOTE.The caller looks like:
do { err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); } while (err == TDX_INTERRUPTED_RESTARTABLE); if (unlikely(tdx_operand_busy(err))) { tdx_no_vcpus_enter_start(kvm); err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); tdx_no_vcpus_enter_stop(kvm); }
Quoted 2 lines
The brute force second case could also be subjected to a
TDX_INTERRUPTED_RESTARTABLEand is not handled. As for interrupt storms, I guess
You are right.
Quoted 3 lines
we could disable interrupts while we do the second brute force case. So the
TDX_INTERRUPTED_RESTARTABLEloop could have a max retries, and the brute force case could also disable interrupts.
Good idea.
Quoted 2 lines
Hmm, how to pick the max retries count. It's a tradeoff between interrupt latency and DOS/code complexity. Do we have any idea how long demote might take?
I did a brief test on my SPR, where the host was not busy :tdh_mem_page_demote() was called 142 times, with each invocation taking around 10us.
2 invocations were due to TDX_INTERRUPTED_RESTARTABLE.
For each GFN, at most 1 retry was performed.
I will do more investigations.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
Quoted 21 lines · Yan Zhao
Quoted 20 lines · Rick P
Quoted 19 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M guest private pages.The "gpa" and "level" direct the TDX module to search and find the old huge leaf entry.
As the new non-leaf entry points to a page table page, callers need to pass in the page table page in parameter "page".
In case of S-EPT walk failure, the entry, level and state where the error was detected are returned in
ext_err1andext_err2.
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
On interrupt pending, SEAMCALL
TDH_MEM_PAGE_DEMOTEreturns errorTDX_INTERRUPTED_RESTARTABLE.[Yan: Rebased and split patch, wrote changelog]
We should add the level of detail here like we did for the base series ones.
I'll provide changelog details under "---" of each patch in the next version.
I mean the commit log (above the "---") needs the same tip style treatment as the other SEAMCALL wrapper patches.
Quoted 32 lines
Quoted 32 lines
Quoted 32 lines
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/include/asm/tdx.h | 2 ++ arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++ arch/x86/virt/vmx/tdx/tdx.h | 1 + 3 files changed, 23 insertions(+)arch/x86/include/asm/tdx.h +2
-0@@ -177,6 +177,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);u64 tdh_mng_create (struct tdx_td *td,u16 hkid);u64 tdh_vp_create (struct tdx_td *td,struct tdx_vp *vp);u64 tdh_mng_rd (struct tdx_td *td,u64 field,u64 *data); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_extend (struct tdx_td *td,u64 gpa,u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_finalize (struct tdx_td *td);u64 tdh_vp_flush (struct tdx_vp *vp);arch/x86/virt/vmx/tdx/tdx.c +0
-0@@ -1684,6 +1684,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) }EXPORT_SYMBOL_GPL (tdh_mng_rd);
Quoted 8 lines
Quoted 7 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1691
Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +5
-0+u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2) +{ +struct tdx_module_args args = { + .rcx = gpa | level,This will only ever be level 2MB, how about dropping the arg?
Do you mean hardcoding level to be 2MB in
tdh_mem_page_demote()?
Yea, we don't support 1GB, so the level arg on the wrapper is superfluous.
Quoted 15 lines
The SEAMCALL
TDH_MEM_PAGE_DEMOTEsupports level of 1GB in current TDX module.Quoted 4 lines
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +4
-0+ .rdx = tdx_tdr_pa(td), + .r8 = page_to_phys(page), + }; +u64 ret;Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1701
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +6
-0+ +tdx_clflush_page (page); + ret =seamcall_ret (TDH_MEM_PAGE_DEMOTE , &args); + + *ext_err1 = args.rcx; + *ext_err2 = args.rdx;How about we just call these entry and
level_state, like the caller.
Quoted 8 lines
Not sure, but I feel that ext_err* might be better, because
- according to the spec,
a) the args.rcx, args.rdx is unmodified (i.e. still hold the passed-in value) in case of errorTDX_INTERRUPTED_RESTARTABLE.
b) args.rcx, args.rdx can only be interpreted as entry andlevel_statein case of EPT walk error.
c) in other cases, they are 0.
- consistent withtdh_mem_page_aug(),tdh_mem_range_block()...
Yea, it's consistent. I'm ok leaving it as is.
Quoted 33 lines
Quoted 26 lines
arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1705
Quoted 6 lines
arch/x86/virt/vmx/tdx/tdx.c +4
-0+ +return ret; +} +EXPORT_SYMBOL_GPL (tdh_mem_page_demote);Looking in the docs, TDX module gives some somewhat constrained guidance:
1. TDH.MEM.PAGE.DEMOTE should be invoked in a loop until it terminates successfully.
2. The host VMM should be designed to avoid cases where interrupt storms prevent successful completion of TDH.MEM.PAGE.DEMOTE.The caller looks like:
do { err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); } while (err == TDX_INTERRUPTED_RESTARTABLE); if (unlikely(tdx_operand_busy(err))) { tdx_no_vcpus_enter_start(kvm); err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state); tdx_no_vcpus_enter_stop(kvm); }Quoted 2 lines
The brute force second case could also be subjected to a
TDX_INTERRUPTED_RESTARTABLEand is not handled. As for interrupt storms, I guessYou are right.
Quoted 3 lines
we could disable interrupts while we do the second brute force case. So the
TDX_INTERRUPTED_RESTARTABLEloop could have a max retries, and the brute force case could also disable interrupts.
Quoted 7 lines
Good idea.
Quoted 2 lines
Hmm, how to pick the max retries count. It's a tradeoff between interrupt latency and DOS/code complexity. Do we have any idea how long demote might take?
I did a brief test on my SPR, where the host was not busy :
tdh_mem_page_demote()was called 142 times, with each invocation taking around 10us.
10us doesn't seem too bad? Makes me think to not loop and instead just do a single retry with interrupts disabled. We should definitely add the data based reasoning to the log.
The counter point is that the SEAMCALL must be supporting TDX_INTERRUPTED_RESTARTABLE for a reason. And the reason probably is that it sometimes takes longer than someone that was reasonable. Maybe we should ask TDX module folks if there is any history.
Quoted 4 lines
2 invocations were due to
TDX_INTERRUPTED_RESTARTABLE. For each GFN, at most 1 retry was performed.I will do more investigations.
Re: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
Quoted 25 lines · Rick P
Quoted 24 lines · Yan Zhao
Quoted 23 lines · Rick P
Quoted 22 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
Add a wrapper
tdh_mem_page_demote()to invoke SEAMCALLTDH_MEM_PAGE_DEMOTEto demote a huge leaf entry to a non-leaf entry in S-EPT. Currently, the TDX module only supports demotion of a 2M huge leaf entry. After a successful demotion, the old 2M huge leaf entry in S-EPT is replaced with a non-leaf entry, linking to the newly-added page table page. The newly linked page table page then contains 512 leaf entries, pointing to the 2M guest private pages.The "gpa" and "level" direct the TDX module to search and find the old huge leaf entry.
As the new non-leaf entry points to a page table page, callers need to pass in the page table page in parameter "page".
In case of S-EPT walk failure, the entry, level and state where the error was detected are returned in
ext_err1andext_err2.On interrupt pending, SEAMCALL
TDH_MEM_PAGE_DEMOTEreturns errorTDX_INTERRUPTED_RESTARTABLE.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
[Yan: Rebased and split patch, wrote changelog]
We should add the level of detail here like we did for the base series ones.
I'll provide changelog details under "---" of each patch in the next version.
I mean the commit log (above the "---") needs the same tip style treatment as the other SEAMCALL wrapper patches.
I thought I have followed the style.
Sorry that if you think the commit msg is too simple without showing details of this SEAMCALL. I can provide a detailed on in the next version if that's the concern you mentioned above.
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 29 lines
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/include/asm/tdx.h | 2 ++ arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++ arch/x86/virt/vmx/tdx/tdx.h | 1 + 3 files changed, 23 insertions(+)arch/x86/include/asm/tdx.h +2
-0@@ -177,6 +177,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);u64 tdh_mng_create (struct tdx_td *td,u16 hkid);u64 tdh_vp_create (struct tdx_td *td,struct tdx_vp *vp);u64 tdh_mng_rd (struct tdx_td *td,u64 field,u64 *data); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_extend (struct tdx_td *td,u64 gpa,u64 *ext_err1,u64 *ext_err2);u64 tdh_mr_finalize (struct tdx_td *td);u64 tdh_vp_flush (struct tdx_vp *vp);arch/x86/virt/vmx/tdx/tdx.c +0
-0arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1691
Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +5
-0@@ -1684,6 +1684,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) }EXPORT_SYMBOL_GPL (tdh_mng_rd); +u64 tdh_mem_page_demote (struct tdx_td *td,u64 gpa,int level,struct page *page, +u64 *ext_err1,u64 *ext_err2) +{ +struct tdx_module_args args = { + .rcx = gpa | level,This will only ever be level 2MB, how about dropping the arg?
Quoted 3 lines
Quoted 1 line
Do you mean hardcoding level to be 2MB in
tdh_mem_page_demote()?Yea, we don't support 1GB, so the level arg on the wrapper is superfluous.
I'm not sure. It's not like tdh_mem_page_add() where the TDX module just only supports 4KB.
But your point that permitting 1GB in tdh_mem_page_demote() in x86 code until after KVM TDX code adds 1GB support also makes sense.
Quoted 12 lines
Quoted 12 lines
The SEAMCALL
TDH_MEM_PAGE_DEMOTEsupports level of 1GB in current TDX module.Quoted 10 lines
Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +10
-0+ .rdx = tdx_tdr_pa(td), + .r8 = page_to_phys(page), + }; +u64 ret; + +tdx_clflush_page (page); + ret =seamcall_ret (TDH_MEM_PAGE_DEMOTE , &args); + + *ext_err1 = args.rcx; + *ext_err2 = args.rdx;
Edgecombe, Rick P
maintainerRe: [RFC PATCH 03/21] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()
In reply to Edgecombe, Rick PQuoted 9 lines · Rick Edgecombe
Quoted 4 lines
I did a brief test on my SPR, where the host was not busy :
tdh_mem_page_demote()was called 142 times, with each invocation taking around
10us.10us doesn't seem too bad? Makes me think to not loop and instead just do a single retry with interrupts disabled. We should definitely add the data based reasoning to the log.
Quoted 5 lines
The counter point is that the SEAMCALL must be supporting
TDX_INTERRUPTED_RESTARTABLEfor a reason. And the reason probably is that it sometimes takes longer than someone that was reasonable. Maybe we should ask TDX
module folks if there is any history.
Circling back here. After some research/discussion it seems demote should not take too long such that it should need the option to return TDX_INTERRUPTED_RESTARTABLE. Even in the dynamic PAMT case. The details of how to get this changed and documented are still ongoing, but for v2 I say we close this by expecting it to never return TDX_INTERRUPTED_RESTARTABLE. For now it can be a VM_BUG_ON() case, with the expectation that TDX module will update to make the logic valid. Sound good?
RFC PATCH 4/21 KVM: TDX: Enforce 4KB mapping level during TD build Time
In reply to Yan ZhaoDuring the TD build phase (i.e., before the TD becomes RUNNABLE), enforce a 4KB mapping level both in the S-EPT managed by the TDX module and the mirror page table managed by KVM.
During this phase, TD's memory is added via tdh_mem_page_add(), which only accepts 4KB granularity. Therefore, return PG_LEVEL_4K in TDX's .private_max_mapping_level hook to ensure KVM maps at the 4KB level in the mirror page table. Meanwhile, iterate over each 4KB page of a large gmem backend page in tdx_gmem_post_populate() and invoke tdh_mem_page_add() to map at the 4KB level in the S-EPT.
Still allow huge pages in gmem backend during TD build time. Based on [1], which gmem series allows 2MB TPH and non-in-place conversion, pass in region.nr_pages to kvm_gmem_populate() in tdx_vcpu_init_mem_region(). This enables kvm_gmem_populate() to allocate huge pages from the gmem backend when the remaining nr_pages, GFN alignment, and page private/shared attribute permit. KVM is then able to promote the initial 4K mapping to huge after TD is RUNNABLE.
Disallow any private huge pages during TD build time. Use BUG_ON() in tdx_mem_page_record_premap_cnt() and tdx_is_sept_zap_err_due_to_premap() to assert the mapping level is 4KB.
Opportunistically, remove unused parameters intdx_mem_page_record_premap_cnt().
--- arch/x86/kvm/vmx/tdx.c | 45 ++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 15 deletions(-)
arch/x86/kvm/vmx/tdx.c +30 -15
@@ -1530,14 +1530,16 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,* The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there * are no half-initialized shared EPT pages. */ -static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn) +static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,enum pg_level level) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EINVAL ; +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return -EINVAL ; +/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);return 0 ;
@@ -1571,7 +1573,7 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE ))return tdx_mem_page_aug (kvm, gfn, level, page); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +return tdx_mem_page_record_premap_cnt (kvm, level); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
@@ -1666,7 +1668,7 @@ int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,static int tdx_is_sept_zap_err_due_to_premap (struct kvm_tdx *kvm_tdx,u64 err,u64 entry,int level) { -if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE ) +if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE || level >PG_LEVEL_4K )return false;if (err != (TDX_EPT_ENTRY_STATE_INCORRECT |TDX_OPERAND_ID_RCX ))
@@ -3052,8 +3054,8 @@ struct tdx_gmem_post_populate_arg {__u32 flags; }; -static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, -void __user *src,int order,void *_arg) +static int tdx_gmem_post_populate_4k (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,void *_arg) {u64 error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS ;struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);
@@ -3120,6 +3122,21 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,return ret; } +static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,int order,void *_arg) +{ +unsigned long i, npages =1 << order; +int ret; + +for (i =0 ; i < npages; i++) { + ret =tdx_gmem_post_populate_4k (kvm, gfn + i, pfn + i, + src + i *PAGE_SIZE , _arg); +if (ret) +return ret; + } +return 0 ; +} +static int tdx_vcpu_init_mem_region (struct kvm_vcpu *vcpu,struct kvm_tdx_cmd *cmd) {struct vcpu_tdx *tdx =to_tdx (vcpu);
@@ -3166,20 +3183,15 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c }; gmem_ret =kvm_gmem_populate (kvm,gpa_to_gfn (region.gpa),u64_to_user_ptr (region.source_addr), -1 , tdx_gmem_post_populate, &arg); + region.nr_pages, tdx_gmem_post_populate, &arg);if (gmem_ret <0 ) { ret = gmem_ret;break ; } -if (gmem_ret !=1 ) { - ret = -EIO ; -break ; - } - - region.source_addr +=PAGE_SIZE ; - region.gpa +=PAGE_SIZE ; - region.nr_pages--; + region.source_addr +=PAGE_SIZE * gmem_ret; + region.gpa +=PAGE_SIZE * gmem_ret; + region.nr_pages -= gmem_ret;cond_resched (); }
@@ -3224,6 +3236,9 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)int tdx_gmem_private_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn) { +if (unlikely (to_kvm_tdx (kvm)->state !=TD_STATE_RUNNABLE )) +return PG_LEVEL_4K ; +return PG_LEVEL_4K ; }
Signature · 1 line
2.43.2
Re: [RFC PATCH 04/21] KVM: TDX: Enforce 4KB mapping level during TD build Time
Quoted 11 lines · Yan Zhao
During the TD build phase (i.e., before the TD becomes RUNNABLE), enforce a 4KB mapping level both in the S-EPT managed by the TDX module and the mirror page table managed by KVM.
During this phase, TD's memory is added via
tdh_mem_page_add(), which only accepts 4KB granularity. Therefore, returnPG_LEVEL_4Kin TDX's .private_max_mapping_level hook to ensure KVM maps at the 4KB level in the mirror page table. Meanwhile, iterate over each 4KB page of a large gmem backend page intdx_gmem_post_populate()and invoketdh_mem_page_add()to map at the 4KB level in the S-EPT.
Quoted 2 lines
Still allow huge pages in gmem backend during TD build time. Based on [1], which gmem series allows 2MB TPH and non-in-place conversion, pass in
s/TPH/THP/
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Re: [RFC PATCH 04/21] KVM: TDX: Enforce 4KB mapping level during TD build Time
Quoted 12 lines · Kirill A. Shutemov
Quoted 11 lines · Yan Zhao
During the TD build phase (i.e., before the TD becomes RUNNABLE), enforce a 4KB mapping level both in the S-EPT managed by the TDX module and the mirror page table managed by KVM.
During this phase, TD's memory is added via
tdh_mem_page_add(), which only accepts 4KB granularity. Therefore, returnPG_LEVEL_4Kin TDX's .private_max_mapping_level hook to ensure KVM maps at the 4KB level in the mirror page table. Meanwhile, iterate over each 4KB page of a large gmem backend page intdx_gmem_post_populate()and invoketdh_mem_page_add()to map at the 4KB level in the S-EPT.
Quoted 4 lines
Quoted 2 lines
Still allow huge pages in gmem backend during TD build time. Based on [1], which gmem series allows 2MB TPH and non-in-place conversion, pass in
s/TPH/THP/
Right. Thanks!
Edgecombe, Rick P
maintainerRe: [RFC PATCH 04/21] KVM: TDX: Enforce 4KB mapping level during TD build Time
In reply to Yan ZhaoQuoted 11 lines · Yan Zhao
During the TD build phase (i.e., before the TD becomes RUNNABLE), enforce a 4KB mapping level both in the S-EPT managed by the TDX module and the mirror page table managed by KVM.
During this phase, TD's memory is added via
tdh_mem_page_add(), which only accepts 4KB granularity. Therefore, returnPG_LEVEL_4Kin TDX's .private_max_mapping_level hook to ensure KVM maps at the 4KB level in the mirror page table. Meanwhile, iterate over each 4KB page of a large gmem backend page intdx_gmem_post_populate()and invoketdh_mem_page_add()to map at the 4KB level in the S-EPT.
Quoted 4 lines
Still allow huge pages in gmem backend during TD build time. Based on [1], which gmem series allows 2MB TPH and non-in-place conversion, pass in region.nr_pages to
kvm_gmem_populate()intdx_vcpu_init_mem_region().
This commit log will need to be written with upstream in mind when it is out of RFC.
Quoted 52 lines
This
enableskvm_gmem_populate()to allocate huge pages from the gmem backend when the remainingnr_pages, GFN alignment, and page private/shared attribute permit. KVM is then able to promote the initial 4K mapping to huge after TD is RUNNABLE.Disallow any private huge pages during TD build time. Use
BUG_ON()intdx_mem_page_record_premap_cnt()andtdx_is_sept_zap_err_due_to_premap()to assert the mapping level is 4KB.Opportunistically, remove unused parameters in
tdx_mem_page_record_premap_cnt().Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 45 ++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 15 deletions(-)arch/x86/kvm/vmx/tdx.c +5
-3@@ -1530,14 +1530,16 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,* The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there * are no half-initialized shared EPT pages. */ -static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn) +static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,enum pg_level level) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EINVAL ; +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return -EINVAL ; +/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);return 0 ;@@ -1571,7 +1573,7 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE ))return tdx_mem_page_aug (kvm, gfn, level, page); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +return tdx_mem_page_record_premap_cnt (kvm, level); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
arch/x86/kvm/vmx/tdx.c int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, · line 1672
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +1
-1@@ -1666,7 +1668,7 @@ int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,static int tdx_is_sept_zap_err_due_to_premap (struct kvm_tdx *kvm_tdx,u64 err,u64 entry,int level) { -if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE ) +if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE || level >PG_LEVEL_4K )return false;
This is catching zapping huge pages before the TD is runnable? Is it necessary if we are already warning about mapping huge pages before the TD is runnable in tdx_mem_page_record_premap_cnt()?
Quoted 45 lines
arch/x86/kvm/vmx/tdx.c +18
-3if (err != (TDX_EPT_ENTRY_STATE_INCORRECT |TDX_OPERAND_ID_RCX ))@@ -3052,8 +3054,8 @@ struct tdx_gmem_post_populate_arg {__u32 flags; }; -static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, -void __user *src,int order,void *_arg) +static int tdx_gmem_post_populate_4k (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,void *_arg) {u64 error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS ;struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);@@ -3120,6 +3122,21 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,return ret; } +static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,int order,void *_arg) +{ +unsigned long i, npages =1 << order; +int ret; + +for (i =0 ; i < npages; i++) { + ret =tdx_gmem_post_populate_4k (kvm, gfn + i, pfn + i, + src + i *PAGE_SIZE , _arg); +if (ret) +return ret; + } +return 0 ; +} +static int tdx_vcpu_init_mem_region (struct kvm_vcpu *vcpu,struct kvm_tdx_cmd *cmd) {struct vcpu_tdx *tdx =to_tdx (vcpu);@@ -3166,20 +3183,15 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c }; gmem_ret =kvm_gmem_populate (kvm,gpa_to_gfn (region.gpa),u64_to_user_ptr (region.source_addr), -1 , tdx_gmem_post_populate, &arg); + region.nr_pages, tdx_gmem_post_populate, &arg);if (gmem_ret <0 ) { ret = gmem_ret;break ; }
arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3192
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +1
-8-if (gmem_ret !=1 ) { - ret = -EIO ; -break ; - } - - region.source_addr +=PAGE_SIZE ; - region.gpa +=PAGE_SIZE ; - region.nr_pages--; + region.source_addr +=PAGE_SIZE * gmem_ret;
gmem_ret has to be 1, per the above conditional.
arch/x86/kvm/vmx/tdx.c int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) · line 3242
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c +5
-0+ region.gpa +=PAGE_SIZE * gmem_ret; + region.nr_pages -= gmem_ret;cond_resched (); }@@ -3224,6 +3236,9 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)int tdx_gmem_private_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn) { +if (unlikely (to_kvm_tdx (kvm)->state !=TD_STATE_RUNNABLE )) +return PG_LEVEL_4K ; +return PG_LEVEL_4K ;
^ Change does nothing...
Quoted 2 lines
arch/x86/kvm/vmx/tdx.c +0
-0}
Re: [RFC PATCH 04/21] KVM: TDX: Enforce 4KB mapping level during TD build Time
Quoted 12 lines · Rick P
Quoted 11 lines · Yan Zhao
During the TD build phase (i.e., before the TD becomes RUNNABLE), enforce a 4KB mapping level both in the S-EPT managed by the TDX module and the mirror page table managed by KVM.
During this phase, TD's memory is added via
tdh_mem_page_add(), which only accepts 4KB granularity. Therefore, returnPG_LEVEL_4Kin TDX's .private_max_mapping_level hook to ensure KVM maps at the 4KB level in the mirror page table. Meanwhile, iterate over each 4KB page of a large gmem backend page intdx_gmem_post_populate()and invoketdh_mem_page_add()to map at the 4KB level in the S-EPT.
Quoted 7 lines
Quoted 4 lines
Still allow huge pages in gmem backend during TD build time. Based on [1], which gmem series allows 2MB TPH and non-in-place conversion, pass in region.nr_pages to
kvm_gmem_populate()intdx_vcpu_init_mem_region().This commit log will need to be written with upstream in mind when it is out of RFC.
Ok.
Quoted 50 lines
Quoted 50 lines
This
enableskvm_gmem_populate()to allocate huge pages from the gmem backend when the remainingnr_pages, GFN alignment, and page private/shared attribute permit. KVM is then able to promote the initial 4K mapping to huge after TD is RUNNABLE.Disallow any private huge pages during TD build time. Use
BUG_ON()intdx_mem_page_record_premap_cnt()andtdx_is_sept_zap_err_due_to_premap()to assert the mapping level is 4KB.Opportunistically, remove unused parameters in
tdx_mem_page_record_premap_cnt().Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 45 ++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 15 deletions(-)arch/x86/kvm/vmx/tdx.c +5
-3@@ -1530,14 +1530,16 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,* The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there * are no half-initialized shared EPT pages. */ -static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn) +static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,enum pg_level level) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EINVAL ; +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return -EINVAL ; +/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);return 0 ;@@ -1571,7 +1573,7 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE ))return tdx_mem_page_aug (kvm, gfn, level, page); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +return tdx_mem_page_record_premap_cnt (kvm, level); }
Quoted 12 lines
arch/x86/kvm/vmx/tdx.c int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, · line 1672
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +1
-1static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,@@ -1666,7 +1668,7 @@ int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,static int tdx_is_sept_zap_err_due_to_premap (struct kvm_tdx *kvm_tdx,u64 err,u64 entry,int level) { -if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE ) +if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE || level >PG_LEVEL_4K )return false;This is catching zapping huge pages before the TD is runnable? Is it necessary if we are already warning about mapping huge pages before the TD is runnable in
tdx_mem_page_record_premap_cnt()?
Under normal conditions, this check isn't necessary.
I added this check in case bugs in the KVM core MMU where the mirror page table might be updated to huge without notifying the TDX side.
Am I overthinking?
Quoted 45 lines
Quoted 45 lines
arch/x86/kvm/vmx/tdx.c +18
-3if (err != (TDX_EPT_ENTRY_STATE_INCORRECT |TDX_OPERAND_ID_RCX ))@@ -3052,8 +3054,8 @@ struct tdx_gmem_post_populate_arg {__u32 flags; }; -static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, -void __user *src,int order,void *_arg) +static int tdx_gmem_post_populate_4k (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,void *_arg) {u64 error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS ;struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);@@ -3120,6 +3122,21 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,return ret; } +static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,int order,void *_arg) +{ +unsigned long i, npages =1 << order; +int ret; + +for (i =0 ; i < npages; i++) { + ret =tdx_gmem_post_populate_4k (kvm, gfn + i, pfn + i, + src + i *PAGE_SIZE , _arg); +if (ret) +return ret; + } +return 0 ; +} +static int tdx_vcpu_init_mem_region (struct kvm_vcpu *vcpu,struct kvm_tdx_cmd *cmd) {struct vcpu_tdx *tdx =to_tdx (vcpu);@@ -3166,20 +3183,15 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c }; gmem_ret =kvm_gmem_populate (kvm,gpa_to_gfn (region.gpa),u64_to_user_ptr (region.source_addr), -1 , tdx_gmem_post_populate, &arg); + region.nr_pages, tdx_gmem_post_populate, &arg);if (gmem_ret <0 ) { ret = gmem_ret;break ; }
Quoted 1 line
arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3175
Quoted 1 line
arch/x86/kvm/vmx/tdx.c +0
-1-if (gmem_ret !=1 ) {
This line is removed.
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3192
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +1
-7- ret = -EIO ; -break ; - } - - region.source_addr +=PAGE_SIZE ; - region.gpa +=PAGE_SIZE ; - region.nr_pages--; + region.source_addr +=PAGE_SIZE * gmem_ret;
gmem_rethas to be 1, per the above conditional.
As region.nr_pages instead of 1 is passed into kvm_gmem_populate(), gmem_ret can now be greater than 1.
kvm_gmem_populate() can allocate huge backend pages if region.nr_pages, GFN alignment, and shareability permit.
Quoted 7 lines
Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +2
-0+ region.gpa +=PAGE_SIZE * gmem_ret; + region.nr_pages -= gmem_ret;cond_resched (); }@@ -3224,6 +3236,9 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) · line 3242
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +3
-0int tdx_gmem_private_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn) { +if (unlikely (to_kvm_tdx (kvm)->state !=TD_STATE_RUNNABLE )) +return PG_LEVEL_4K ; +return PG_LEVEL_4K ;^ Change does nothing...
Right. Patch 9 will update the default level to PG_LEVEL_2M.
The change here is meant to highlight PG_LEVEL_4K is enforced in tdx_gmem_private_max_mapping_level() when TD is not in TD_STATE_RUNNABLE state.
Will change it in the next version.
Quoted 3 lines
Quoted 2 lines
arch/x86/kvm/vmx/tdx.c +0
-0}
Edgecombe, Rick P
maintainerRe: [RFC PATCH 04/21] KVM: TDX: Enforce 4KB mapping level during TD build Time
Quoted 85 lines · Yan Zhao
Quoted 12 lines · Rick P
Quoted 11 lines · Yan Zhao
During the TD build phase (i.e., before the TD becomes RUNNABLE), enforce a 4KB mapping level both in the S-EPT managed by the TDX module and the mirror page table managed by KVM.
During this phase, TD's memory is added via
tdh_mem_page_add(), which only accepts 4KB granularity. Therefore, returnPG_LEVEL_4Kin TDX's .private_max_mapping_level hook to ensure KVM maps at the 4KB level in the mirror page table. Meanwhile, iterate over each 4KB page of a large gmem backend page intdx_gmem_post_populate()and invoketdh_mem_page_add()to map at the 4KB level in the S-EPT.Quoted 7 lines
Quoted 4 lines
Still allow huge pages in gmem backend during TD build time. Based on [1], which gmem series allows 2MB TPH and non-in-place conversion, pass in region.nr_pages to
kvm_gmem_populate()intdx_vcpu_init_mem_region().This commit log will need to be written with upstream in mind when it is out of RFC.
Ok.
Quoted 50 lines
Quoted 50 lines
This
enableskvm_gmem_populate()to allocate huge pages from the gmem backend when the remainingnr_pages, GFN alignment, and page private/shared attribute permit. KVM is then able to promote the initial 4K mapping to huge after TD is RUNNABLE.Disallow any private huge pages during TD build time. Use
BUG_ON()intdx_mem_page_record_premap_cnt()andtdx_is_sept_zap_err_due_to_premap()to assert the mapping level is 4KB.Opportunistically, remove unused parameters in
tdx_mem_page_record_premap_cnt().Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 45 ++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 15 deletions(-)arch/x86/kvm/vmx/tdx.c +5
-3@@ -1530,14 +1530,16 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,* The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there * are no half-initialized shared EPT pages. */ -static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn) +static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,enum pg_level level) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EINVAL ; +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return -EINVAL ; +/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);return 0 ;@@ -1571,7 +1573,7 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE ))return tdx_mem_page_aug (kvm, gfn, level, page); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +return tdx_mem_page_record_premap_cnt (kvm, level); }Quoted 12 lines
arch/x86/kvm/vmx/tdx.c int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, · line 1672
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +1
-1static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,@@ -1666,7 +1668,7 @@ int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,static int tdx_is_sept_zap_err_due_to_premap (struct kvm_tdx *kvm_tdx,u64 err,u64 entry,int level) { -if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE ) +if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE || level >PG_LEVEL_4K )return false;This is catching zapping huge pages before the TD is runnable? Is it necessary if we are already warning about mapping huge pages before the TD is runnable in
tdx_mem_page_record_premap_cnt()?
Quoted 4 lines
Under normal conditions, this check isn't necessary.
I added this check in case bugs in the KVM core MMU where the mirror page table might be updated to huge without notifying the TDX side.
Am I overthinking?
If we need so many BUG_ON()s maybe our design is too fragile. I think we could drop this one.
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
arch/x86/kvm/vmx/tdx.c +18
-3if (err != (TDX_EPT_ENTRY_STATE_INCORRECT |TDX_OPERAND_ID_RCX ))@@ -3052,8 +3054,8 @@ struct tdx_gmem_post_populate_arg {__u32 flags; }; -static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, -void __user *src,int order,void *_arg) +static int tdx_gmem_post_populate_4k (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,void *_arg) {u64 error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS ;struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);@@ -3120,6 +3122,21 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,return ret; } +static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,int order,void *_arg) +{ +unsigned long i, npages =1 << order; +int ret; + +for (i =0 ; i < npages; i++) { + ret =tdx_gmem_post_populate_4k (kvm, gfn + i, pfn + i, + src + i *PAGE_SIZE , _arg); +if (ret) +return ret; + } +return 0 ; +} +static int tdx_vcpu_init_mem_region (struct kvm_vcpu *vcpu,struct kvm_tdx_cmd *cmd) {struct vcpu_tdx *tdx =to_tdx (vcpu);@@ -3166,20 +3183,15 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c }; gmem_ret =kvm_gmem_populate (kvm,gpa_to_gfn (region.gpa),u64_to_user_ptr (region.source_addr), -1 , tdx_gmem_post_populate, &arg); + region.nr_pages, tdx_gmem_post_populate, &arg);if (gmem_ret <0 ) { ret = gmem_ret;break ; }
Quoted 2 lines
Quoted 1 line
arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3175
Quoted 1 line
arch/x86/kvm/vmx/tdx.c +0
-1-if (gmem_ret !=1 ) {This line is removed.
Doh! Right.
Quoted 32 lines
Quoted 11 lines
arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3192
Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +1
-7- ret = -EIO ; -break ; - } - - region.source_addr +=PAGE_SIZE ; - region.gpa +=PAGE_SIZE ; - region.nr_pages--; + region.source_addr +=PAGE_SIZE * gmem_ret;
gmem_rethas to be 1, per the above conditional.As region.nr_pages instead of 1 is passed into
kvm_gmem_populate(),gmem_retcan now be greater than 1.
kvm_gmem_populate()can allocate huge backend pages if region.nr_pages, GFN alignment, and shareability permit.Quoted 7 lines
Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +2
-0+ region.gpa +=PAGE_SIZE * gmem_ret; + region.nr_pages -= gmem_ret;cond_resched (); }@@ -3224,6 +3236,9 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)Quoted 8 lines
arch/x86/kvm/vmx/tdx.c int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) · line 3242
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +3
-0int tdx_gmem_private_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn) { +if (unlikely (to_kvm_tdx (kvm)->state !=TD_STATE_RUNNABLE )) +return PG_LEVEL_4K ; +return PG_LEVEL_4K ;^ Change does nothing...
Quoted 6 lines
Right. Patch 9 will update the default level to
PG_LEVEL_2M.The change here is meant to highlight
PG_LEVEL_4Kis enforced intdx_gmem_private_max_mapping_level()when TD is not inTD_STATE_RUNNABLEstate.Will change it in the next version.
I can't see the pattern between what goes in this patch vs patch 9. We should have some reasoning behind it.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +0
-0}
Re: [RFC PATCH 04/21] KVM: TDX: Enforce 4KB mapping level during TD build Time
Quoted 86 lines · Rick P
Quoted 85 lines · Yan Zhao
Quoted 12 lines · Rick P
Quoted 11 lines · Yan Zhao
During the TD build phase (i.e., before the TD becomes RUNNABLE), enforce a 4KB mapping level both in the S-EPT managed by the TDX module and the mirror page table managed by KVM.
During this phase, TD's memory is added via
tdh_mem_page_add(), which only accepts 4KB granularity. Therefore, returnPG_LEVEL_4Kin TDX's .private_max_mapping_level hook to ensure KVM maps at the 4KB level in the mirror page table. Meanwhile, iterate over each 4KB page of a large gmem backend page intdx_gmem_post_populate()and invoketdh_mem_page_add()to map at the 4KB level in the S-EPT.Quoted 7 lines
Quoted 4 lines
Still allow huge pages in gmem backend during TD build time. Based on [1], which gmem series allows 2MB TPH and non-in-place conversion, pass in region.nr_pages to
kvm_gmem_populate()intdx_vcpu_init_mem_region().This commit log will need to be written with upstream in mind when it is out of RFC.
Ok.
Quoted 62 lines
Quoted 52 lines
This
enableskvm_gmem_populate()to allocate huge pages from the gmem backend when the remainingnr_pages, GFN alignment, and page private/shared attribute permit. KVM is then able to promote the initial 4K mapping to huge after TD is RUNNABLE.Disallow any private huge pages during TD build time. Use
BUG_ON()intdx_mem_page_record_premap_cnt()andtdx_is_sept_zap_err_due_to_premap()to assert the mapping level is 4KB.Opportunistically, remove unused parameters in
tdx_mem_page_record_premap_cnt().Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 45 ++++++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 15 deletions(-)arch/x86/kvm/vmx/tdx.c +5
-3@@ -1530,14 +1530,16 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,* The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there * are no half-initialized shared EPT pages. */ -static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,gfn_t gfn, -enum pg_level level,kvm_pfn_t pfn) +static int tdx_mem_page_record_premap_cnt (struct kvm *kvm,enum pg_level level) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);if (KVM_BUG_ON (kvm->arch.pre_fault_allowed, kvm))return -EINVAL ; +if (KVM_BUG_ON (level !=PG_LEVEL_4K , kvm)) +return -EINVAL ; +/* nr_premapped will be decreased when tdh_mem_page_add() is called. */ atomic64_inc (&kvm_tdx->nr_premapped);return 0 ;@@ -1571,7 +1573,7 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,if (likely (kvm_tdx->state ==TD_STATE_RUNNABLE ))return tdx_mem_page_aug (kvm, gfn, level, page); -return tdx_mem_page_record_premap_cnt (kvm, gfn, level, pfn); +return tdx_mem_page_record_premap_cnt (kvm, level); } static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,arch/x86/kvm/vmx/tdx.c int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn, · line 1672
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +1
-1@@ -1666,7 +1668,7 @@ int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,static int tdx_is_sept_zap_err_due_to_premap (struct kvm_tdx *kvm_tdx,u64 err,u64 entry,int level) { -if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE ) +if (!err || kvm_tdx->state ==TD_STATE_RUNNABLE || level >PG_LEVEL_4K )return false;This is catching zapping huge pages before the TD is runnable? Is it necessary if we are already warning about mapping huge pages before the TD is runnable in
tdx_mem_page_record_premap_cnt()?
Quoted 7 lines
Quoted 4 lines
Under normal conditions, this check isn't necessary.
I added this check in case bugs in the KVM core MMU where the mirror page table might be updated to huge without notifying the TDX side.
Am I overthinking?If we need so many BUG_ON()s maybe our design is too fragile. I think we could drop this one.
Got it.
Quoted 86 lines
Quoted 47 lines
Quoted 47 lines
Quoted 47 lines
arch/x86/kvm/vmx/tdx.c +18
-3if (err != (TDX_EPT_ENTRY_STATE_INCORRECT |TDX_OPERAND_ID_RCX ))@@ -3052,8 +3054,8 @@ struct tdx_gmem_post_populate_arg {__u32 flags; }; -static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, -void __user *src,int order,void *_arg) +static int tdx_gmem_post_populate_4k (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,void *_arg) {u64 error_code =PFERR_GUEST_FINAL_MASK |PFERR_PRIVATE_ACCESS ;struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);@@ -3120,6 +3122,21 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,return ret; } +static int tdx_gmem_post_populate (struct kvm *kvm,gfn_t gfn,kvm_pfn_t pfn, +void __user *src,int order,void *_arg) +{ +unsigned long i, npages =1 << order; +int ret; + +for (i =0 ; i < npages; i++) { + ret =tdx_gmem_post_populate_4k (kvm, gfn + i, pfn + i, + src + i *PAGE_SIZE , _arg); +if (ret) +return ret; + } +return 0 ; +} +static int tdx_vcpu_init_mem_region (struct kvm_vcpu *vcpu,struct kvm_tdx_cmd *cmd) {struct vcpu_tdx *tdx =to_tdx (vcpu);@@ -3166,20 +3183,15 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c }; gmem_ret =kvm_gmem_populate (kvm,gpa_to_gfn (region.gpa),u64_to_user_ptr (region.source_addr), -1 , tdx_gmem_post_populate, &arg); + region.nr_pages, tdx_gmem_post_populate, &arg);if (gmem_ret <0 ) { ret = gmem_ret;break ; }Quoted 2 lines
Quoted 1 line
arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3175
Quoted 1 line
arch/x86/kvm/vmx/tdx.c +0
-1-if (gmem_ret !=1 ) {This line is removed.
Doh! Right.
Quoted 34 lines
Quoted 11 lines
arch/x86/kvm/vmx/tdx.c static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c · line 3192
Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +1
-7- ret = -EIO ; -break ; - } - - region.source_addr +=PAGE_SIZE ; - region.gpa +=PAGE_SIZE ; - region.nr_pages--; + region.source_addr +=PAGE_SIZE * gmem_ret;
gmem_rethas to be 1, per the above conditional.As region.nr_pages instead of 1 is passed into
kvm_gmem_populate(),gmem_retcan now be greater than 1.
kvm_gmem_populate()can allocate huge backend pages if region.nr_pages, GFN alignment, and shareability permit.Quoted 7 lines
Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +2
-0+ region.gpa +=PAGE_SIZE * gmem_ret; + region.nr_pages -= gmem_ret;cond_resched (); }@@ -3224,6 +3236,9 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)Quoted 8 lines
arch/x86/kvm/vmx/tdx.c int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp) · line 3242
Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +3
-0int tdx_gmem_private_max_mapping_level (struct kvm *kvm,kvm_pfn_t pfn) { +if (unlikely (to_kvm_tdx (kvm)->state !=TD_STATE_RUNNABLE )) +return PG_LEVEL_4K ; +return PG_LEVEL_4K ;^ Change does nothing...
Right. Patch 9 will update the default level to
PG_LEVEL_2M.
Quoted 7 lines
Quoted 4 lines
The change here is meant to highlight
PG_LEVEL_4Kis enforced intdx_gmem_private_max_mapping_level()when TD is not inTD_STATE_RUNNABLEstate.Will change it in the next version.
I can't see the pattern between what goes in this patch vs patch 9. We should have some reasoning behind it.
tdx_gmem_private_max_mapping_level() actually has no need to change in this patch as both cases return PG_LEVEL_4K.
My reasoning behind it was that I hoped to show the whole picture to support TDX huge page before TD is runnable in this patch as much as possible.
Without this superfluous modification, the info that PG_LEVEL_4K is returned before TD is runnable would not be noticed in this patch.
It's definitely reasonable for the later versions to drop the hunk of the change to tdx_gmem_private_max_mapping_level().
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +0
-0}
RFC PATCH 5/21 KVM: TDX: Enhance tdx_clear_page() to support huge pages
In reply to Yan ZhaoFrom: Xiaoyao Li <[email protected]>
KVM invokes tdx_clear_page() to zero pages using movdir64b(). Include level information to enable tdx_clear_page() to zero a huge page.
[Yan: split out, let tdx_clear_page() accept level]
--- arch/x86/kvm/vmx/tdx.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/tdx.c +14 -5
@@ -276,7 +276,7 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu) vcpu->cpu =-1 ; } -static void tdx_clear_page (struct page *page) +static void __tdx_clear_page (struct page *page) {const void *zero_page = (const void *)page_to_virt (ZERO_PAGE (0 ));void *dest =page_to_virt (page);
@@ -295,6 +295,15 @@ static void tdx_clear_page(struct page *page)__mb (); } +static void tdx_clear_page (struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; + +while (nr--) +__tdx_clear_page (nth_page (page, idx++)); +} +static void tdx_no_vcpus_enter_start (struct kvm *kvm) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);
@@ -340,11 +349,10 @@ static int tdx_reclaim_page(struct page *page) r =__tdx_reclaim_page (page);if (!r) -tdx_clear_page (page); +tdx_clear_page (page,PG_LEVEL_4K );return r; } -/* * Reclaim the TD control page(s) which are crypto-protected by TDX guest's * private KeyID. Assume the cache associated with the TDX private KeyID has
@@ -588,7 +596,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm)pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err);return ; } -tdx_clear_page (kvm_tdx->td.tdr_page); +tdx_clear_page (kvm_tdx->td.tdr_page,PG_LEVEL_4K );__free_page (kvm_tdx->td.tdr_page); kvm_tdx->td.tdr_page =NULL ;
@@ -1621,7 +1629,8 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err);return -EIO ; } -tdx_clear_page (page); + +tdx_clear_page (page, level);tdx_unpin (kvm, page);return 0 ; }
Signature · 1 line
2.43.2
Edgecombe, Rick P
maintainerRe: [RFC PATCH 05/21] KVM: TDX: Enhance tdx_clear_page() to support huge pages
Quoted 28 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
KVM invokes
tdx_clear_page()to zero pages usingmovdir64b(). Include level information to enabletdx_clear_page()to zero a huge page.[Yan: split out, let
tdx_clear_page()accept level]Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/tdx.c +1
-1@@ -276,7 +276,7 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu) vcpu->cpu =-1 ; } -static void tdx_clear_page (struct page *page) +static void __tdx_clear_page (struct page *page) {const void *zero_page = (const void *)page_to_virt (ZERO_PAGE (0 ));void *dest =page_to_virt (page);
arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 304
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-0@@ -295,6 +295,15 @@ static void tdx_clear_page(struct page *page)__mb (); } +static void tdx_clear_page (struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; + +while (nr--) +__tdx_clear_page (nth_page (page, idx++));
You shouldn't need both idx and nr.
arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 305
Quoted 1 line
arch/x86/kvm/vmx/tdx.c +1
-0+}
Since tdx_clear_page() has a __mb(), it is probably worth checking that this generates efficient code, considering the loops within loops pattern.
Quoted 18 lines
arch/x86/kvm/vmx/tdx.c +2
-2+static void tdx_no_vcpus_enter_start (struct kvm *kvm) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);@@ -340,11 +349,10 @@ static int tdx_reclaim_page(struct page *page) r =__tdx_reclaim_page (page);if (!r) -tdx_clear_page (page); +tdx_clear_page (page,PG_LEVEL_4K );return r; } -/* * Reclaim the TD control page(s) which are crypto-protected by TDX guest's * private KeyID. Assume the cache associated with the TDX private KeyID has
arch/x86/kvm/vmx/tdx.c static void tdx_reclaim_td_control_pages(struct kvm *kvm) · line 599
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +1
-1@@ -588,7 +596,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm)pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err);return ; } -tdx_clear_page (kvm_tdx->td.tdr_page); +tdx_clear_page (kvm_tdx->td.tdr_page,PG_LEVEL_4K );
Why not the __tdx_clear_page() variant? The patch adds it, but doesn't really use it. Just implement it all in tdx_clear_page() then.
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c +2
-1__free_page (kvm_tdx->td.tdr_page); kvm_tdx->td.tdr_page =NULL ;@@ -1621,7 +1629,8 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err);return -EIO ; } -tdx_clear_page (page); + +tdx_clear_page (page, level);tdx_unpin (kvm, page);return 0 ; }
Re: [RFC PATCH 05/21] KVM: TDX: Enhance tdx_clear_page() to support huge pages
Quoted 42 lines · Rick P
Quoted 28 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
KVM invokes
tdx_clear_page()to zero pages usingmovdir64b(). Include level information to enabletdx_clear_page()to zero a huge page.[Yan: split out, let
tdx_clear_page()accept level]Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/tdx.c +1
-1@@ -276,7 +276,7 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu) vcpu->cpu =-1 ; } -static void tdx_clear_page (struct page *page) +static void __tdx_clear_page (struct page *page) {const void *zero_page = (const void *)page_to_virt (ZERO_PAGE (0 ));void *dest =page_to_virt (page);arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 304
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7
-0@@ -295,6 +295,15 @@ static void tdx_clear_page(struct page *page)__mb (); } +static void tdx_clear_page (struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; + +while (nr--) +__tdx_clear_page (nth_page (page, idx++));You shouldn't need both idx and nr.
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 305
Quoted 1 line
arch/x86/kvm/vmx/tdx.c +1
-0+}Since
tdx_clear_page()has a__mb(), it is probably worth checking that this generates efficient code, considering the loops within loops pattern.
The concern makes sense!
Will convert level to size and use "for (i = 0; i < size; i += 64)" for movdir64b().
Quoted 13 lines
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c +2
-1+static void tdx_no_vcpus_enter_start (struct kvm *kvm) {struct kvm_tdx *kvm_tdx =to_kvm_tdx (kvm);@@ -340,11 +349,10 @@ static int tdx_reclaim_page(struct page *page) r =__tdx_reclaim_page (page);if (!r) -tdx_clear_page (page); +tdx_clear_page (page,PG_LEVEL_4K );return r; }
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c static void tdx_reclaim_td_control_pages(struct kvm *kvm) · line 599
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +1
-2-/* * Reclaim the TD control page(s) which are crypto-protected by TDX guest's * private KeyID. Assume the cache associated with the TDX private KeyID has @@ -588,7 +596,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm)pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err);return ; } -tdx_clear_page (kvm_tdx->td.tdr_page); +tdx_clear_page (kvm_tdx->td.tdr_page,PG_LEVEL_4K );Why not the
__tdx_clear_page()variant? The patch adds it, but doesn't really use it. Just implement it all intdx_clear_page()then.
Ok.
Quoted 14 lines
Quoted 13 lines
arch/x86/kvm/vmx/tdx.c +2
-1__free_page (kvm_tdx->td.tdr_page); kvm_tdx->td.tdr_page =NULL ;@@ -1621,7 +1629,8 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err);return -EIO ; } -tdx_clear_page (page); + +tdx_clear_page (page, level);tdx_unpin (kvm, page);return 0 ; }
RFC PATCH 6/21 KVM: TDX: Assert the reclaimed pages were mapped as expected
In reply to Yan ZhaoFrom: Xiaoyao Li <[email protected]>
Provide level information to tdx_reclaim_page() to enable it to verify that the reclaimed pages were mapped at the expected level in the S-EPT.
[Yan: split patch, wrote patch log]
--- arch/x86/kvm/vmx/tdx.c | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-)
arch/x86/kvm/vmx/tdx.c +9 -7
@@ -325,7 +325,7 @@ static void tdx_no_vcpus_enter_stop(struct kvm *kvm) }/* TDH.PHYMEM.PAGE.RECLAIM is allowed only when destroying the TD. */ -static int __tdx_reclaim_page (struct page *page) +static int __tdx_reclaim_page (struct page *page,int level) {u64 err, tdx_pt, tdx_owner, tdx_size;
@@ -340,16 +340,18 @@ static int __tdx_reclaim_page(struct page *page)pr_tdx_error_3 (TDH_PHYMEM_PAGE_RECLAIM , err, tdx_pt, tdx_owner, tdx_size);return -EIO ; } + +WARN_ON_ONCE (tdx_size !=pg_level_to_tdx_sept_level (level));return 0 ; } -static int tdx_reclaim_page (struct page *page) +static int tdx_reclaim_page (struct page *page,int level) {int r; - r =__tdx_reclaim_page (page); + r =__tdx_reclaim_page (page, level);if (!r) -tdx_clear_page (page,PG_LEVEL_4K ); +tdx_clear_page (page, level);return r; }
@@ -364,7 +366,7 @@ static void tdx_reclaim_control_page(struct page *ctrl_page)* Leak the page if the kernel failed to reclaim the page. * The kernel cannot use it safely anymore. */ -if (tdx_reclaim_page (ctrl_page)) +if (tdx_reclaim_page (ctrl_page,PG_LEVEL_4K ))return ;__free_page (ctrl_page);
@@ -583,7 +585,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm)if (!kvm_tdx->td.tdr_page)return ; -if (__tdx_reclaim_page (kvm_tdx->td.tdr_page)) +if (__tdx_reclaim_page (kvm_tdx->td.tdr_page,PG_LEVEL_4K ))return ;/*
@@ -1791,7 +1793,7 @@ int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,* The HKID assigned to this TD was already freed and cache was * already flushed. We don't have to flush again. */ -return tdx_reclaim_page (virt_to_page (private_spt)); +return tdx_reclaim_page (virt_to_page (private_spt),PG_LEVEL_4K ); } int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
Signature · 1 line
2.43.2
Edgecombe, Rick P
maintainerRe: [RFC PATCH 06/21] KVM: TDX: Assert the reclaimed pages were mapped as expected
arch/x86/kvm/vmx/tdx.c static int __tdx_reclaim_page(struct page *page) · line 344
Quoted 12 lines · Yan Zhao
/* TDH.PHYMEM.PAGE.RECLAIM is allowed only when destroying the TD. */
arch/x86/kvm/vmx/tdx.c +1
-1-static int __tdx_reclaim_page (struct page *page) +static int __tdx_reclaim_page (struct page *page,int level){
u64 err,tdx_pt,tdx_owner,tdx_size;arch/x86/kvm/vmx/tdx.c +2
-0@@ -340,16 +340,18 @@ static int __tdx_reclaim_page(struct page *page) pr_tdx_error_3(TDH_PHYMEM_PAGE_RECLAIM, err, tdx_pt, tdx_owner, tdx_size); return -EIO; } + +WARN_ON_ONCE (tdx_size !=pg_level_to_tdx_sept_level (level));
Why not return an error in this case?
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +0
-0return 0; }
No callers in the series pass anything other than PG_LEVEL_4K, so do we need this patch?
Re: [RFC PATCH 06/21] KVM: TDX: Assert the reclaimed pages were mapped as expected
Quoted 15 lines · Rick P
arch/x86/kvm/vmx/tdx.c static int __tdx_reclaim_page(struct page *page) · line 344
Quoted 12 lines · Yan Zhao
/* TDH.PHYMEM.PAGE.RECLAIM is allowed only when destroying the TD. */
arch/x86/kvm/vmx/tdx.c +1
-1-static int __tdx_reclaim_page (struct page *page) +static int __tdx_reclaim_page (struct page *page,int level){
u64 err,tdx_pt,tdx_owner,tdx_size;arch/x86/kvm/vmx/tdx.c +2
-0@@ -340,16 +340,18 @@ static int __tdx_reclaim_page(struct page *page) pr_tdx_error_3(TDH_PHYMEM_PAGE_RECLAIM, err, tdx_pt, tdx_owner, tdx_size); return -EIO; } + +WARN_ON_ONCE (tdx_size !=pg_level_to_tdx_sept_level (level));Why not return an error in this case?
Yes, returing error seems reasonable, which indicate a series bug.
Quoted 6 lines
Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +0
-0return 0; }No callers in the series pass anything other than
PG_LEVEL_4K, so do we need this patch?
Oh, this patch is only for future VM shutdown optimization where huge guest pages could be reclaimed.
We can of couse include it in the VM shutdown optimization series if you think it's better.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 06/21] KVM: TDX: Assert the reclaimed pages were mapped as expected
Quoted 6 lines · Yan Zhao
Quoted 2 lines
No callers in the series pass anything other than
PG_LEVEL_4K, so do we need this patch?Oh, this patch is only for future VM shutdown optimization where huge guest pages could be reclaimed.
We can of couse include it in the VM shutdown optimization series if you think it's better.
I think it's better.
RFC PATCH 7/21 KVM: TDX: Add a helper for WBINVD on huge pages with TD's keyID
In reply to Yan ZhaoFrom: Xiaoyao Li <[email protected]>
After a guest page is removed from the S-EPT, KVM calls tdh_phymem_page_wbinvd_hkid() to execute WBINVD on the page using the TD's keyID.
Add a helper function that takes level information to perform WBINVD on a huge page.
--- arch/x86/kvm/vmx/tdx.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)
arch/x86/kvm/vmx/tdx.c +19 -5
@@ -1586,6 +1586,23 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return tdx_mem_page_record_premap_cnt (kvm, level); } +static inline u64 tdx_wbinvd_page (struct kvm *kvm,u64 hkid,struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; +u64 err; + +while (nr--) { + err =tdh_phymem_page_wbinvd_hkid (hkid,nth_page (page, idx++)); + +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); +return err; + } + } +return err; +} +static int tdx_sept_drop_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {
@@ -1625,12 +1642,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; } - err =tdh_phymem_page_wbinvd_hkid ((u16 )kvm_tdx->hkid, page); - -if (KVM_BUG_ON (err, kvm)) { -pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); + err =tdx_wbinvd_page (kvm, kvm_tdx->hkid, page, level); +if (err)return -EIO ; - }tdx_clear_page (page, level);tdx_unpin (kvm, page);
Signature · 1 line
2.43.2
Re: [RFC PATCH 07/21] KVM: TDX: Add a helper for WBINVD on huge pages with TD's keyID
Quoted 46 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
After a guest page is removed from the S-EPT, KVM calls
tdh_phymem_page_wbinvd_hkid()to execute WBINVD on the page using the TD's keyID.Add a helper function that takes level information to perform WBINVD on a huge page.
[Yan: split patch, added a helper, rebased to use struct page]Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/tdx.c +17
-0@@ -1586,6 +1586,23 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return tdx_mem_page_record_premap_cnt (kvm, level); } +static inline u64 tdx_wbinvd_page (struct kvm *kvm,u64 hkid,struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; +u64 err; + +while (nr--) { + err =tdh_phymem_page_wbinvd_hkid (hkid,nth_page (page, idx++)); + +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); +return err; + } + } +return err; +} +static int tdx_sept_drop_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {
arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1646
Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +2
-4@@ -1625,12 +1642,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; } - err =tdh_phymem_page_wbinvd_hkid ((u16 )kvm_tdx->hkid, page); - -if (KVM_BUG_ON (err, kvm)) { -pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); + err =tdx_wbinvd_page (kvm, kvm_tdx->hkid, page, level); +if (err)
It can add unlikely() here.
Also the err is not used after check, maybe it can be combined as:
if (unlikely(tdx_wbinvd_page(kvm, kvm_tdx->hkid, page, level))) return -EIO;
Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +0
-1return -EIO ; - }tdx_clear_page (page, level);tdx_unpin (kvm, page);
Re: [RFC PATCH 07/21] KVM: TDX: Add a helper for WBINVD on huge pages with TD's keyID
Quoted 60 lines · Binbin Wu
Quoted 45 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
After a guest page is removed from the S-EPT, KVM calls
tdh_phymem_page_wbinvd_hkid()to execute WBINVD on the page using the TD's keyID.Add a helper function that takes level information to perform WBINVD on a huge page.
[Yan: split patch, added a helper, rebased to use struct page]Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/tdx.c +17
-0@@ -1586,6 +1586,23 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return tdx_mem_page_record_premap_cnt (kvm, level); } +static inline u64 tdx_wbinvd_page (struct kvm *kvm,u64 hkid,struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; +u64 err; + +while (nr--) { + err =tdh_phymem_page_wbinvd_hkid (hkid,nth_page (page, idx++)); + +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); +return err; + } + } +return err; +} +static int tdx_sept_drop_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1645
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +2
-4@@ -1625,12 +1642,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; } - err =tdh_phymem_page_wbinvd_hkid ((u16 )kvm_tdx->hkid, page); - -if (KVM_BUG_ON (err, kvm)) { -pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); + err =tdx_wbinvd_page (kvm, kvm_tdx->hkid, page, level); +if (err)It can add
unlikely()here.
Also the err is not used after check, maybe it can be combined as:
Quoted 2 lines
if (
unlikely(tdx_wbinvd_page(kvm, kvm_tdx->hkid, page, level))) return-EIO;
That's better. Thanks!
Quoted 5 lines
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +0
-1return -EIO ; - }tdx_clear_page (page, level);tdx_unpin (kvm, page);
Edgecombe, Rick P
maintainerRe: [RFC PATCH 07/21] KVM: TDX: Add a helper for WBINVD on huge pages with TD's keyID
In reply to Yan ZhaoQuoted 31 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
After a guest page is removed from the S-EPT, KVM calls
tdh_phymem_page_wbinvd_hkid()to execute WBINVD on the page using the TD's keyID.Add a helper function that takes level information to perform WBINVD on a huge page.
[Yan: split patch, added a helper, rebased to use struct page]Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/tdx.c +6
-0@@ -1586,6 +1586,23 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return tdx_mem_page_record_premap_cnt (kvm, level); } +static inline u64 tdx_wbinvd_page (struct kvm *kvm,u64 hkid,struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; +u64 err; +
arch/x86/kvm/vmx/tdx.c int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1604
Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10
-0+while (nr--) { + err =tdh_phymem_page_wbinvd_hkid (hkid,nth_page (page, idx++)); + +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); +return err; + } + } +return err; +}
Hmm, did you consider changing tdh_phymem_page_wbinvd_hkid()? It's the pattern of KVM wrapping the SEAMCALL helpers to do some more work that needs to be wrapped.
Quoted 19 lines
arch/x86/kvm/vmx/tdx.c +3
-5+static int tdx_sept_drop_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {@@ -1625,12 +1642,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; } - err =tdh_phymem_page_wbinvd_hkid ((u16 )kvm_tdx->hkid, page); - -if (KVM_BUG_ON (err, kvm)) { -pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); + err =tdx_wbinvd_page (kvm, kvm_tdx->hkid, page, level); +if (err)return -EIO ; - }tdx_clear_page (page, level);tdx_unpin (kvm, page);
Re: [RFC PATCH 07/21] KVM: TDX: Add a helper for WBINVD on huge pages with TD's keyID
Quoted 38 lines · Rick P
Quoted 37 lines · Yan Zhao
From: Xiaoyao Li <[email protected]>
After a guest page is removed from the S-EPT, KVM calls
tdh_phymem_page_wbinvd_hkid()to execute WBINVD on the page using the TD's keyID.Add a helper function that takes level information to perform WBINVD on a huge page.
[Yan: split patch, added a helper, rebased to use struct page]Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-)arch/x86/kvm/vmx/tdx.c +12
-0@@ -1586,6 +1586,23 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,return tdx_mem_page_record_premap_cnt (kvm, level); } +static inline u64 tdx_wbinvd_page (struct kvm *kvm,u64 hkid,struct page *page,int level) +{ +unsigned long nr =KVM_PAGES_PER_HPAGE (level); +unsigned long idx =0 ; +u64 err; + +while (nr--) { + err =tdh_phymem_page_wbinvd_hkid (hkid,nth_page (page, idx++)); + +if (KVM_BUG_ON (err, kvm)) { +pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); +return err;
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1604
Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4
-0+ } + } +return err; +}Hmm, did you consider changing
tdh_phymem_page_wbinvd_hkid()? It's the pattern of KVM wrapping the SEAMCALL helpers to do some more work that needs to be wrapped.
SEAMCALL TDH_PHYMEM_PAGE_WBINVD only accepts a 4KB page.
Will move the loop from KVM to the wrapper in x86 if you think it's better.
Quoted 19 lines
Quoted 18 lines
arch/x86/kvm/vmx/tdx.c +2
-5static int tdx_sept_drop_private_spte (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {@@ -1625,12 +1642,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ; } - err =tdh_phymem_page_wbinvd_hkid ((u16 )kvm_tdx->hkid, page); - -if (KVM_BUG_ON (err, kvm)) { -pr_tdx_error (TDH_PHYMEM_PAGE_WBINVD , err); + err =tdx_wbinvd_page (kvm, kvm_tdx->hkid, page, level); +if (err)return -EIO ; - }tdx_clear_page (page, level);tdx_unpin (kvm, page);
Edgecombe, Rick P
maintainerRe: [RFC PATCH 07/21] KVM: TDX: Add a helper for WBINVD on huge pages with TD's keyID
Quoted 6 lines · Yan Zhao
Quoted 4 lines
Hmm, did you consider changing
tdh_phymem_page_wbinvd_hkid()? It's the pattern
of KVM wrapping the SEAMCALL helpers to do some more work that needs to be wrapped.SEAMCALL
TDH_PHYMEM_PAGE_WBINVDonly accepts a 4KB page.
Will move the loop from KVM to the wrapper in x86 if you think it's better.
Don't wrap the wrappers was a suggestion from Dave. Let's try it.
RFC PATCH 8/21 KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Yan ZhaoIncrease folio ref count before mapping a private page, and decrease folio ref count after a mapping failure or successfully removing a private page.
The folio ref count to inc/dec corresponds to the mapping/unmapping level, ensuring the folio ref count remains balanced after entry splitting or merging.
--- arch/x86/kvm/vmx/tdx.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
arch/x86/kvm/vmx/tdx.c +10 -9
@@ -1501,9 +1501,9 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) +static void tdx_unpin (struct kvm *kvm,struct page *page,int level) { -put_page (page); +folio_put_refs (page_folio (page),KVM_PAGES_PER_HPAGE (level)); } static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
@@ -1517,13 +1517,13 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state);if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EBUSY ; }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EIO ; }
@@ -1570,10 +1570,11 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* a_ops->migrate_folio (yet), no callback is triggered for KVM on page * migration. Until guest_memfd supports page migration, prevent page * migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). +* TODO: To support in-place-conversion in gmem in futre, remove +* folio_ref_add()/folio_put_refs(). Only increase the folio ref count +* when there're errors during removing private pages. */ -get_page (page); +folio_ref_add (page_folio (page),KVM_PAGES_PER_HPAGE (level));/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching
@@ -1647,7 +1648,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;tdx_clear_page (page, level); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; }
@@ -1727,7 +1728,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; }
Signature · 1 line
2.43.2
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 49 lines · Yan Zhao
Increase folio ref count before mapping a private page, and decrease folio ref count after a mapping failure or successfully removing a private page.
The folio ref count to inc/dec corresponds to the mapping/unmapping level, ensuring the folio ref count remains balanced after entry splitting or merging.
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-4@@ -1501,9 +1501,9 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) +static void tdx_unpin (struct kvm *kvm,struct page *page,int level) { -put_page (page); +folio_put_refs (page_folio (page),KVM_PAGES_PER_HPAGE (level)); } static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,@@ -1517,13 +1517,13 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state);if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EBUSY ; }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EIO ; }
arch/x86/kvm/vmx/tdx.c int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1574
Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +2
-2@@ -1570,10 +1570,11 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* a_ops->migrate_folio (yet), no callback is triggered for KVM on page * migration. Until guest_memfd supports page migration, prevent page * migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). +* TODO: To support in-place-conversion in gmem in futre, remove +* folio_ref_add()/folio_put_refs().
With necessary infrastructure in guest_memfd [1] to prevent page migration, is it necessary to acquire extra folio refcounts? If not, why not just cleanup this logic now?
[1] https://git.kernel.org/pub/scm/virt/kvm/kvm.git/tree/virt/kvm/guest_memfd.c?h=kvm-coco-queue#n441
Quoted 29 lines
Only increase the folio ref count
arch/x86/kvm/vmx/tdx.c +4
-4+* when there're errors during removing private pages. */ -get_page (page); +folio_ref_add (page_folio (page),KVM_PAGES_PER_HPAGE (level));/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching @@ -1647,7 +1648,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;tdx_clear_page (page, level); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; }@@ -1727,7 +1728,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; } --2.43.2
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 49 lines · Vishal Annapurve
Quoted 48 lines · Yan Zhao
Increase folio ref count before mapping a private page, and decrease folio ref count after a mapping failure or successfully removing a private page.
The folio ref count to inc/dec corresponds to the mapping/unmapping level, ensuring the folio ref count remains balanced after entry splitting or merging.
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-4@@ -1501,9 +1501,9 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) +static void tdx_unpin (struct kvm *kvm,struct page *page,int level) { -put_page (page); +folio_put_refs (page_folio (page),KVM_PAGES_PER_HPAGE (level)); } static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,@@ -1517,13 +1517,13 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state);if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EBUSY ; }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EIO ; }
Quoted 12 lines
arch/x86/kvm/vmx/tdx.c int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1574
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +2
-2@@ -1570,10 +1570,11 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* a_ops->migrate_folio (yet), no callback is triggered for KVM on page * migration. Until guest_memfd supports page migration, prevent page * migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). +* TODO: To support in-place-conversion in gmem in futre, remove +* folio_ref_add()/folio_put_refs(). With necessary infrastructure in
guest_memfd[1] to prevent page migration, is it necessary to acquire extra folio refcounts? If not, why not just cleanup this logic now?
Though the old comment says acquiring the lock is for page migration, the other reason is to prevent the folio from being returned to the OS until it has been successfully removed from TDX.
If there's an error during the removal or reclaiming of a folio from TDX, such as a failure in tdh_mem_page_remove()/tdh_phymem_page_wbinvd_hkid() or tdh_phymem_page_reclaim(), it is important to hold the page refcount within TDX.
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking folio_ref_add() in the event of a removal failure.
Quoted 31 lines
[1] https://git.kernel.org/pub/scm/virt/kvm/kvm.git/tree/virt/kvm/guest_memfd.c?h=kvm-coco-queue#n441
Quoted 29 lines
Only increase the folio ref count
arch/x86/kvm/vmx/tdx.c +4
-4+* when there're errors during removing private pages. */ -get_page (page); +folio_ref_add (page_folio (page),KVM_PAGES_PER_HPAGE (level));/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching @@ -1647,7 +1648,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;tdx_clear_page (page, level); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; }@@ -1727,7 +1728,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; } --2.43.2
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 67 lines · Yan Zhao
Quoted 49 lines · Vishal Annapurve
Quoted 48 lines · Yan Zhao
Increase folio ref count before mapping a private page, and decrease folio ref count after a mapping failure or successfully removing a private page.
The folio ref count to inc/dec corresponds to the mapping/unmapping level, ensuring the folio ref count remains balanced after entry splitting or merging.
Signed-off-by Xiaoyao Li <[email protected]>Signed-off-by Isaku Yamahata <[email protected]>Signed-off-by Yan Zhao <[email protected]>--- arch/x86/kvm/vmx/tdx.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)arch/x86/kvm/vmx/tdx.c +4
-4@@ -1501,9 +1501,9 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } -static void tdx_unpin (struct kvm *kvm,struct page *page) +static void tdx_unpin (struct kvm *kvm,struct page *page,int level) { -put_page (page); +folio_put_refs (page_folio (page),KVM_PAGES_PER_HPAGE (level)); } static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,@@ -1517,13 +1517,13 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn, err =tdh_mem_page_aug (&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state);if (unlikely (tdx_operand_busy (err))) { -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EBUSY ; }if (KVM_BUG_ON (err, kvm)) {pr_tdx_error_2 (TDH_MEM_PAGE_AUG , err, entry, level_state); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return -EIO ; }Quoted 12 lines
arch/x86/kvm/vmx/tdx.c int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn, · line 1574
Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +2
-2@@ -1570,10 +1570,11 @@ int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,* a_ops->migrate_folio (yet), no callback is triggered for KVM on page * migration. Until guest_memfd supports page migration, prevent page * migration. -* TODO: Once guest_memfd introduces callback on page migration, -* implement it and remove get_page/put_page(). +* TODO: To support in-place-conversion in gmem in futre, remove +* folio_ref_add()/folio_put_refs(). With necessary infrastructure in
guest_memfd[1] to prevent page migration, is it necessary to acquire extra folio refcounts? If not, why not just cleanup this logic now?Though the old comment says acquiring the lock is for page migration, the other reason is to prevent the folio from being returned to the OS until it has been successfully removed from TDX.
Quoted 6 lines
If there's an error during the removal or reclaiming of a folio from TDX, such as a failure in tdh_mem_page_remove()/tdh_phymem_page_wbinvd_hkid() or
tdh_phymem_page_reclaim(), it is important to hold the page refcount within TDX.So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.
In my opinion, the above scheme can be deployed with this series itself. guest_memfd will not take away memory from TDX VMs without an invalidation. folio_ref_add() will not work for memory not backed by page structs, but that problem can be solved in future possibly by notifying guest_memfd of certain ranges being in use even after invalidation completes.
Quoted 32 lines
Quoted 32 lines
[1] https://git.kernel.org/pub/scm/virt/kvm/kvm.git/tree/virt/kvm/guest_memfd.c?h=kvm-coco-queue#n441
Quoted 29 lines
Only increase the folio ref count
arch/x86/kvm/vmx/tdx.c +4
-4+* when there're errors during removing private pages. */ -get_page (page); +folio_ref_add (page_folio (page),KVM_PAGES_PER_HPAGE (level));/* * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching @@ -1647,7 +1648,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,return -EIO ;tdx_clear_page (page, level); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; }@@ -1727,7 +1728,7 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,if (tdx_is_sept_zap_err_due_to_premap (kvm_tdx, err, entry, level) && !KVM_BUG_ON (!atomic64_read (&kvm_tdx->nr_premapped), kvm)) {atomic64_dec (&kvm_tdx->nr_premapped); -tdx_unpin (kvm, page); +tdx_unpin (kvm, page, level);return 0 ; } --2.43.2
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Sorry for the late reply, I was on leave last week.
Quoted 6 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without an
I initially intended to add a separate patch at the end of this series to implement invoking folio_ref_add() only upon a removal failure. However, I decided against it since it's not a must before guest_memfd supports in-place conversion.
We can include it in the next version If you think it's better.
Quoted 2 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly by
With current TDX code, all memory must be backed by a page struct. Both tdh_mem_page_add() and tdh_mem_page_aug() require a "struct page *" rather than a pfn.
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.
A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.
Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.
Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?
I imagine the needed support will match similar semantics as VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.
Any guest_memfd range updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring to guest_memfd backed pfns. This story will become clearer once the support for PFN range allocator for backing guest_memfd starts getting discussed.
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 37 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?
Quoted 4 lines
I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.
So, it's possible to map such memory in both shared and private EPT simultaneously?
Quoted 5 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.
Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of VM_PFNMAP memory into private EPT until TDX connect.
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than guest_memfd.
Quoted 1 line
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 39 lines · Yan Zhao
Quoted 37 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 13 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?
Quoted 6 lines
Quoted 4 lines
I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.So, it's possible to map such memory in both shared and private EPT simultaneously?
No, guest_memfd will still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.
There is a plan to use VM_PFNMAP memory for all of guest_memfd shared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be that guest_memfd private ranges will be mapped into IOMMU page tables.
Irrespective of whether/when VM_PFNMAP memory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts for guest_memfd private ranges and/or page structs altogether.
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for guest_memfd as well.
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/
Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.
Not necessary. As I mentioned above guest_memfd ranges will be backed by VM_PFNMAP memory.
Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 44 lines · Vishal Annapurve
Quoted 43 lines · Yan Zhao
Quoted 41 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.
Quoted 5 lines
Quoted 2 lines
So, it's possible to map such memory in both shared and private EPT simultaneously?
No,
guest_memfdwill still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.
Before guest_memfd converts a PFN from shared to private, how does it ensure there are no shared mappings? e.g., in [1], it uses the folio reference count to ensure that.
Or do you believe that by eliminating the struct page, there would be no GUP, thereby ensuring no shared mappings by requiring all mappers to unmap in response to a guest_memfd invalidation notification?
As in Documentation/core-api/pin_user_pages.rst, long-term pinning users have no need to register mmu notifier. So why users like VFIO must register guest_memfd invalidation notification?
Besides, how would guest_memfd handle potential unmap failures? e.g. what happens to prevent converting a private PFN to shared if there are errors when TDX unmaps a private PFN or if a device refuses to stop DMAing to a PFN.
Currently, guest_memfd can rely on page ref count to avoid re-assigning a PFN that fails to be unmapped.
[1] https://lore.kernel.org/all/[email protected]/
Quoted 17 lines
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.There is a plan to use
VM_PFNMAPmemory for all ofguest_memfdshared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be thatguest_memfdprivate ranges will be mapped into IOMMU page tables.
Quoted 7 lines
Irrespective of whether/when
VM_PFNMAPmemory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts forguest_memfdprivate ranges and/or page structs altogether.
It's fine to use PFN, but I wonder if there're counterparts of struct page to keep all necessary info.
Quoted 3 lines
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for
guest_memfdas well.
Note: Currently, VM_PFNMAP is usually used together with flag VM_IO. in KVM hva_to_pfn_remapped() only applies to "vma->vm_flags & (VM_IO | VM_PFNMAP)".
Quoted 12 lines
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.Not necessary. As I mentioned above
guest_memfdranges will be backed byVM_PFNMAPmemory.Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 51 lines · Yan Zhao
Quoted 44 lines · Vishal Annapurve
Quoted 43 lines · Yan Zhao
Quoted 41 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.Quoted 5 lines
Quoted 2 lines
So, it's possible to map such memory in both shared and private EPT simultaneously?
No,
guest_memfdwill still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.
Quoted 7 lines
Before
guest_memfdconverts a PFN from shared to private, how does it ensure there are no shared mappings? e.g., in [1], it uses the folio reference count to ensure that.Or do you believe that by eliminating the struct page, there would be no GUP, thereby ensuring no shared mappings by requiring all mappers to unmap in response to a
guest_memfdinvalidation notification?
Yes.
Quoted 4 lines
As in
Documentation/core-api/pin_user_pages.rst, long-term pinning users have no need to register mmu notifier. So why users like VFIO must registerguest_memfdinvalidation notification?
VM_PFNMAP'd memory can't be long term pinned, so users of such memory ranges will have to adopt mechanisms to get notified. I think it would be easy to pursue new users of guest_memfd to follow this scheme. Irrespective of whether VM_PFNMAP'd support lands, guest_memfd hugepage support already needs the stance of: "Guest memfd owns all long-term refcounts on private memory" as discussed at LPC [1].
[1] https://lpc.events/event/18/contributions/1764/attachments/1409/3182/LPC%202024_%201G%20page%20support%20for%20guest_memfd.pdf
(slide 12)
Quoted 4 lines
Besides, how would
guest_memfdhandle potential unmap failures? e.g. what happens to prevent converting a private PFN to shared if there are errors when TDX unmaps a private PFN or if a device refuses to stop DMAing to a PFN.
Users will have to signal such failures via the invalidation callback results or other appropriate mechanisms. guest_memfd can relay the failures up the call chain to the userspace.
Quoted 32 lines
Currently,
guest_memfdcan rely on page ref count to avoid re-assigning a PFN that fails to be unmapped.[1] https://lore.kernel.org/all/[email protected]/
Quoted 17 lines
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.There is a plan to use
VM_PFNMAPmemory for all ofguest_memfdshared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be thatguest_memfdprivate ranges will be mapped into IOMMU page tables.Quoted 7 lines
Irrespective of whether/when
VM_PFNMAPmemory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts forguest_memfdprivate ranges and/or page structs altogether.
Quoted 3 lines
It's fine to use PFN, but I wonder if there're counterparts of struct page to keep all necessary info.
Story will become clearer once VM_PFNMAP'd memory support starts getting discussed. In case of guest_memfd, there is flexibility to store metadata for physical ranges within guest_memfd just like shareability tracking.
Quoted 20 lines
Quoted 4 lines
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for
guest_memfdas well.Note: Currently,
VM_PFNMAPis usually used together with flagVM_IO. in KVMhva_to_pfn_remapped()only applies to "vma->vm_flags& (VM_IO|VM_PFNMAP)".Quoted 12 lines
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.Not necessary. As I mentioned above
guest_memfdranges will be backed byVM_PFNMAPmemory.Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 63 lines · Vishal Annapurve
Quoted 51 lines · Yan Zhao
Quoted 44 lines · Vishal Annapurve
Quoted 43 lines · Yan Zhao
Quoted 41 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.Quoted 5 lines
Quoted 2 lines
So, it's possible to map such memory in both shared and private EPT simultaneously?
No,
guest_memfdwill still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.Quoted 7 lines
Before
guest_memfdconverts a PFN from shared to private, how does it ensure there are no shared mappings? e.g., in [1], it uses the folio reference count to ensure that.Or do you believe that by eliminating the struct page, there would be no GUP, thereby ensuring no shared mappings by requiring all mappers to unmap in response to a
guest_memfdinvalidation notification?Yes.
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
As in
Documentation/core-api/pin_user_pages.rst, long-term pinning users have no need to register mmu notifier. So why users like VFIO must registerguest_memfdinvalidation notification?VM_PFNMAP'd memory can't be long term pinned, so users of such memory ranges will have to adopt mechanisms to get notified. I think it would
Hmm, in current VFIO, it does not register any notifier for VM_PFNMAP'd memory.
Quoted 9 lines
be easy to pursue new users of
guest_memfdto follow this scheme. Irrespective of whether VM_PFNMAP'd support lands,guest_memfdhugepage support already needs the stance of: "Guest memfd owns all long-term refcounts on private memory" as discussed at LPC [1].[1] https://lpc.events/event/18/contributions/1764/attachments/1409/3182/LPC%202024_%201G%20page%20support%20for%20guest_memfd.pdf
(slide 12)Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Besides, how would
guest_memfdhandle potential unmap failures? e.g. what happens to prevent converting a private PFN to shared if there are errors when TDX unmaps a private PFN or if a device refuses to stop DMAing to a PFN.Users will have to signal such failures via the invalidation callback results or other appropriate mechanisms.
guest_memfdcan relay the failures up the call chain to the userspace.
AFAIK, operations that perform actual unmapping do not allow failure, e.g. kvm_mmu_unmap_gfn_range(), iopt_area_unfill_domains(), vfio_iommu_unmap_unpin_all(), vfio_iommu_unmap_unpin_reaccount().
That's why we rely on increasing folio ref count to reflect failure, which are due to unexpected SEAMCALL errors.
Quoted 31 lines
Quoted 31 lines
Currently,
guest_memfdcan rely on page ref count to avoid re-assigning a PFN that fails to be unmapped.[1] https://lore.kernel.org/all/[email protected]/
Quoted 24 lines
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.There is a plan to use
VM_PFNMAPmemory for all ofguest_memfdshared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be thatguest_memfdprivate ranges will be mapped into IOMMU page tables.Irrespective of whether/when
VM_PFNMAPmemory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts forguest_memfdprivate ranges and/or page structs altogether.
Quoted 8 lines
Quoted 3 lines
It's fine to use PFN, but I wonder if there're counterparts of struct page to keep all necessary info.
Story will become clearer once VM_PFNMAP'd memory support starts getting discussed. In case of
guest_memfd, there is flexibility to store metadata for physical ranges withinguest_memfdjust like shareability tracking.
Ok.
Quoted 21 lines
Quoted 20 lines
Quoted 4 lines
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for
guest_memfdas well.Note: Currently,
VM_PFNMAPis usually used together with flagVM_IO. in KVMhva_to_pfn_remapped()only applies to "vma->vm_flags& (VM_IO|VM_PFNMAP)".Quoted 12 lines
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.Not necessary. As I mentioned above
guest_memfdranges will be backed byVM_PFNMAPmemory.Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 65 lines · Yan Zhao
Quoted 63 lines · Vishal Annapurve
Quoted 51 lines · Yan Zhao
Quoted 44 lines · Vishal Annapurve
Quoted 43 lines · Yan Zhao
Quoted 41 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.Quoted 5 lines
Quoted 2 lines
So, it's possible to map such memory in both shared and private EPT simultaneously?
No,
guest_memfdwill still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.Quoted 7 lines
Before
guest_memfdconverts a PFN from shared to private, how does it ensure there are no shared mappings? e.g., in [1], it uses the folio reference count to ensure that.Or do you believe that by eliminating the struct page, there would be no GUP, thereby ensuring no shared mappings by requiring all mappers to unmap in response to a
guest_memfdinvalidation notification?Yes.
Quoted 1 line
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
As in
Documentation/core-api/pin_user_pages.rst, long-term pinning users have no need to register mmu notifier. So why users like VFIO must registerguest_memfdinvalidation notification?VM_PFNMAP'd memory can't be long term pinned, so users of such memory ranges will have to adopt mechanisms to get notified. I think it would
Hmm, in current VFIO, it does not register any notifier for VM_PFNMAP'd memory.
I don't completely understand how VM_PFNMAP'd memory is used today for VFIO. Maybe only MMIO regions are backed by pfnmap today and the story for normal memory backed by pfnmap is yet to materialize.
Quoted 14 lines
Quoted 14 lines
be easy to pursue new users of
guest_memfdto follow this scheme. Irrespective of whether VM_PFNMAP'd support lands,guest_memfdhugepage support already needs the stance of: "Guest memfd owns all long-term refcounts on private memory" as discussed at LPC [1].[1] https://lpc.events/event/18/contributions/1764/attachments/1409/3182/LPC%202024_%201G%20page%20support%20for%20guest_memfd.pdf
(slide 12)Quoted 4 lines
Besides, how would
guest_memfdhandle potential unmap failures? e.g. what happens to prevent converting a private PFN to shared if there are errors when TDX unmaps a private PFN or if a device refuses to stop DMAing to a PFN.
Quoted 6 lines
Quoted 3 lines
Users will have to signal such failures via the invalidation callback results or other appropriate mechanisms.
guest_memfdcan relay the failures up the call chain to the userspace.AFAIK, operations that perform actual unmapping do not allow failure, e.g.
kvm_mmu_unmap_gfn_range(),iopt_area_unfill_domains(),vfio_iommu_unmap_unpin_all(),vfio_iommu_unmap_unpin_reaccount().
Very likely because these operations simply don't fail.
Quoted 3 lines
That's why we rely on increasing folio ref count to reflect failure, which are due to unexpected SEAMCALL errors.
TDX stack is adding a scenario where invalidation can fail, a cleaner solution would be to propagate the result as an invalidation failure. Another option is to notify guest_memfd out of band to convey the ranges that failed invalidation.
With in-place conversion supported, even if the refcount is raised for such pages, they can still get used by the host if the guest_memfd is unaware that the invalidation failed.
Quoted 63 lines
Quoted 32 lines
Quoted 32 lines
Currently,
guest_memfdcan rely on page ref count to avoid re-assigning a PFN that fails to be unmapped.[1] https://lore.kernel.org/all/[email protected]/
Quoted 24 lines
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.There is a plan to use
VM_PFNMAPmemory for all ofguest_memfdshared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be thatguest_memfdprivate ranges will be mapped into IOMMU page tables.Irrespective of whether/when
VM_PFNMAPmemory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts forguest_memfdprivate ranges and/or page structs altogether.Quoted 8 lines
Quoted 3 lines
It's fine to use PFN, but I wonder if there're counterparts of struct page to keep all necessary info.
Story will become clearer once VM_PFNMAP'd memory support starts getting discussed. In case of
guest_memfd, there is flexibility to store metadata for physical ranges withinguest_memfdjust like shareability tracking.Ok.
Quoted 21 lines
Quoted 20 lines
Quoted 4 lines
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for
guest_memfdas well.Note: Currently,
VM_PFNMAPis usually used together with flagVM_IO. in KVMhva_to_pfn_remapped()only applies to "vma->vm_flags& (VM_IO|VM_PFNMAP)".Quoted 12 lines
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.Not necessary. As I mentioned above
guest_memfdranges will be backed byVM_PFNMAPmemory.Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 70 lines · Vishal Annapurve
Quoted 69 lines · Yan Zhao
Quoted 67 lines · Vishal Annapurve
Quoted 51 lines · Yan Zhao
Quoted 44 lines · Vishal Annapurve
Quoted 43 lines · Yan Zhao
Quoted 41 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.Quoted 5 lines
Quoted 2 lines
So, it's possible to map such memory in both shared and private EPT simultaneously?
No,
guest_memfdwill still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.Quoted 7 lines
Before
guest_memfdconverts a PFN from shared to private, how does it ensure there are no shared mappings? e.g., in [1], it uses the folio reference count to ensure that.Or do you believe that by eliminating the struct page, there would be no GUP, thereby ensuring no shared mappings by requiring all mappers to unmap in response to a
guest_memfdinvalidation notification?Yes.
Quoted 4 lines
As in
Documentation/core-api/pin_user_pages.rst, long-term pinning users have no need to register mmu notifier. So why users like VFIO must registerguest_memfdinvalidation notification?
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
VM_PFNMAP'd memory can't be long term pinned, so users of such memory ranges will have to adopt mechanisms to get notified. I think it would
Hmm, in current VFIO, it does not register any notifier for VM_PFNMAP'd memory.
I don't completely understand how VM_PFNMAP'd memory is used today for VFIO. Maybe only MMIO regions are backed by pfnmap today and the story for normal memory backed by pfnmap is yet to materialize.
VFIO can fault in VM_PFNMAP'd memory which is not from MMIO regions. It works because it knows VM_PFNMAP'd memory are always pinned.
Another example is udmabuf (drivers/dma-buf/udmabuf.c), it mmaps normal folios with VM_PFNMAP flag without registering mmu notifier because those folios are pinned.
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
be easy to pursue new users of
guest_memfdto follow this scheme. Irrespective of whether VM_PFNMAP'd support lands,guest_memfdhugepage support already needs the stance of: "Guest memfd owns all long-term refcounts on private memory" as discussed at LPC [1].[1] https://lpc.events/event/18/contributions/1764/attachments/1409/3182/LPC%202024_%201G%20page%20support%20for%20guest_memfd.pdf
(slide 12)Quoted 4 lines
Besides, how would
guest_memfdhandle potential unmap failures? e.g. what happens to prevent converting a private PFN to shared if there are errors when TDX unmaps a private PFN or if a device refuses to stop DMAing to a PFN.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Users will have to signal such failures via the invalidation callback results or other appropriate mechanisms.
guest_memfdcan relay the failures up the call chain to the userspace.AFAIK, operations that perform actual unmapping do not allow failure, e.g.
kvm_mmu_unmap_gfn_range(),iopt_area_unfill_domains(),vfio_iommu_unmap_unpin_all(),vfio_iommu_unmap_unpin_reaccount().Very likely because these operations simply don't fail.
I think they are intentionally designed to be no-fail.
e.g. in __iopt_area_unfill_domain(), no-fail is achieved by using a small backup buffer allocated on stack in case of kmalloc() failure.
Quoted 6 lines
Quoted 3 lines
That's why we rely on increasing folio ref count to reflect failure, which are due to unexpected SEAMCALL errors.
TDX stack is adding a scenario where invalidation can fail, a cleaner solution would be to propagate the result as an invalidation failure.
Not sure if linux kernel accepts unmap failure.
Quoted 2 lines
Another option is to notify
guest_memfdout of band to convey the ranges that failed invalidation.
Yes, this might be better. Something similar like holding folio ref count to let guest_memfd know that a certain PFN cannot be re-assigned.
Quoted 3 lines
With in-place conversion supported, even if the refcount is raised for such pages, they can still get used by the host if the
guest_memfdis unaware that the invalidation failed.
I thought guest_memfd should check if folio ref count is 0 (or a base count) before conversion, splitting or re-assignment. Otherwise, why do you care if TDX holds the ref count? :)
Quoted 63 lines
Quoted 63 lines
Quoted 32 lines
Quoted 32 lines
Currently,
guest_memfdcan rely on page ref count to avoid re-assigning a PFN that fails to be unmapped.[1] https://lore.kernel.org/all/[email protected]/
Quoted 24 lines
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.There is a plan to use
VM_PFNMAPmemory for all ofguest_memfdshared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be thatguest_memfdprivate ranges will be mapped into IOMMU page tables.Irrespective of whether/when
VM_PFNMAPmemory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts forguest_memfdprivate ranges and/or page structs altogether.Quoted 8 lines
Quoted 3 lines
It's fine to use PFN, but I wonder if there're counterparts of struct page to keep all necessary info.
Story will become clearer once VM_PFNMAP'd memory support starts getting discussed. In case of
guest_memfd, there is flexibility to store metadata for physical ranges withinguest_memfdjust like shareability tracking.Ok.
Quoted 21 lines
Quoted 20 lines
Quoted 4 lines
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for
guest_memfdas well.Note: Currently,
VM_PFNMAPis usually used together with flagVM_IO. in KVMhva_to_pfn_remapped()only applies to "vma->vm_flags& (VM_IO|VM_PFNMAP)".Quoted 12 lines
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.Not necessary. As I mentioned above
guest_memfdranges will be backed byVM_PFNMAPmemory.Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 79 lines · Yan Zhao
Quoted 70 lines · Vishal Annapurve
Quoted 69 lines · Yan Zhao
Quoted 67 lines · Vishal Annapurve
Quoted 51 lines · Yan Zhao
Quoted 44 lines · Vishal Annapurve
Quoted 43 lines · Yan Zhao
Quoted 41 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.Quoted 5 lines
Quoted 2 lines
So, it's possible to map such memory in both shared and private EPT simultaneously?
No,
guest_memfdwill still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.Quoted 7 lines
Before
guest_memfdconverts a PFN from shared to private, how does it ensure there are no shared mappings? e.g., in [1], it uses the folio reference count to ensure that.Or do you believe that by eliminating the struct page, there would be no GUP, thereby ensuring no shared mappings by requiring all mappers to unmap in response to a
guest_memfdinvalidation notification?Yes.
Quoted 4 lines
As in
Documentation/core-api/pin_user_pages.rst, long-term pinning users have no need to register mmu notifier. So why users like VFIO must registerguest_memfdinvalidation notification?Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
VM_PFNMAP'd memory can't be long term pinned, so users of such memory ranges will have to adopt mechanisms to get notified. I think it would
Hmm, in current VFIO, it does not register any notifier for VM_PFNMAP'd memory.
I don't completely understand how VM_PFNMAP'd memory is used today for VFIO. Maybe only MMIO regions are backed by pfnmap today and the story for normal memory backed by pfnmap is yet to materialize.
Quoted 7 lines
VFIO can fault in VM_PFNMAP'd memory which is not from MMIO regions. It works because it knows VM_PFNMAP'd memory are always pinned.
Another example is udmabuf (
drivers/dma-buf/udmabuf.c), it mmaps normal folios withVM_PFNMAPflag without registering mmu notifier because those folios are pinned.
I might be wrongly throwing out some terminologies here then. VM_PFNMAP flag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.
The goal is to get to a stage where guest_memfd is backed by pfn ranges unmanaged by kernel that guest_memfd owns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if the shareability changes, the users will get notified and will have to invalidate their mappings. guest_memfd will allow mmaping such ranges with VM_PFNMAP flag set by default in the VMAs to indicate the need of special handling/lack of page structs.
As an intermediate stage, it makes sense to me to just not have private memory backed by page structs and use a special "filemap" to map file offsets to these private memory ranges. This step will also need similar contract with users -
1) memory is pinned by guest_memfd
2) users will get invalidation notifiers on shareability changes
I am sure there is a lot of work here and many quirks to be addressed, let's discuss this more with better context around. A few related RFC series are planned to be posted in the near future.
Quoted 42 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
be easy to pursue new users of
guest_memfdto follow this scheme. Irrespective of whether VM_PFNMAP'd support lands,guest_memfdhugepage support already needs the stance of: "Guest memfd owns all long-term refcounts on private memory" as discussed at LPC [1].[1] https://lpc.events/event/18/contributions/1764/attachments/1409/3182/LPC%202024_%201G%20page%20support%20for%20guest_memfd.pdf
(slide 12)Quoted 4 lines
Besides, how would
guest_memfdhandle potential unmap failures? e.g. what happens to prevent converting a private PFN to shared if there are errors when TDX unmaps a private PFN or if a device refuses to stop DMAing to a PFN.Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Users will have to signal such failures via the invalidation callback results or other appropriate mechanisms.
guest_memfdcan relay the failures up the call chain to the userspace.AFAIK, operations that perform actual unmapping do not allow failure, e.g.
kvm_mmu_unmap_gfn_range(),iopt_area_unfill_domains(),vfio_iommu_unmap_unpin_all(),vfio_iommu_unmap_unpin_reaccount().Very likely because these operations simply don't fail.
I think they are intentionally designed to be no-fail.
e.g. in
__iopt_area_unfill_domain(), no-fail is achieved by using a small backup buffer allocated on stack in case ofkmalloc()failure.Quoted 6 lines
Quoted 3 lines
That's why we rely on increasing folio ref count to reflect failure, which are due to unexpected SEAMCALL errors.
TDX stack is adding a scenario where invalidation can fail, a cleaner solution would be to propagate the result as an invalidation failure.
Not sure if linux kernel accepts unmap failure.
Quoted 2 lines
Another option is to notify
guest_memfdout of band to convey the ranges that failed invalidation.Yes, this might be better. Something similar like holding folio ref count to let
guest_memfdknow that a certain PFN cannot be re-assigned.
Quoted 7 lines
Quoted 3 lines
With in-place conversion supported, even if the refcount is raised for such pages, they can still get used by the host if the
guest_memfdis unaware that the invalidation failed.I thought
guest_memfdshould check if folio ref count is 0 (or a base count) before conversion, splitting or re-assignment. Otherwise, why do you care if TDX holds the ref count? :)
Soon to be posted RFC series by Ackerley currently explicitly checks for safe private page refcounts when folio splitting is needed and not for every private to shared conversion. A simple solution would be for guest_memfd to check safe page refcounts for each private to shared conversion even if split is not required but will need to be reworked when either of the stages discussed above land where page structs are not around.
Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 33 lines
Quoted 33 lines
Currently,
guest_memfdcan rely on page ref count to avoid re-assigning a PFN that fails to be unmapped.[1] https://lore.kernel.org/all/[email protected]/
Quoted 24 lines
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.There is a plan to use
VM_PFNMAPmemory for all ofguest_memfdshared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be thatguest_memfdprivate ranges will be mapped into IOMMU page tables.Irrespective of whether/when
VM_PFNMAPmemory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts forguest_memfdprivate ranges and/or page structs altogether.Quoted 8 lines
Quoted 3 lines
It's fine to use PFN, but I wonder if there're counterparts of struct page to keep all necessary info.
Story will become clearer once VM_PFNMAP'd memory support starts getting discussed. In case of
guest_memfd, there is flexibility to store metadata for physical ranges withinguest_memfdjust like shareability tracking.Ok.
Quoted 21 lines
Quoted 20 lines
Quoted 4 lines
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for
guest_memfdas well.Note: Currently,
VM_PFNMAPis usually used together with flagVM_IO. in KVMhva_to_pfn_remapped()only applies to "vma->vm_flags& (VM_IO|VM_PFNMAP)".Quoted 12 lines
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.Not necessary. As I mentioned above
guest_memfdranges will be backed byVM_PFNMAPmemory.Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 11 lines · Vishal Annapurve
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if the shareability changes, the users will get notified and will have to invalidate their mappings.guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.
I see the point about how operating on PFNs can allow smoother transition to a solution that saves struct page memory, but I wonder about the wisdom of building this 2MB TDX code against eventual goals.
We were thinking to enable 2MB TDX huge pages on top of:
1. Mmap shared pages
2. In-place conversion
3. 2MB huge page support
Where do you think struct page-less guestmemfd fits in that roadmap?
Quoted 11 lines
As an intermediate stage, it makes sense to me to just not have private memory backed by page structs and use a special "filemap" to map file offsets to these private memory ranges. This step will also need similar contract with users -
1) memory is pinned byguest_memfd
2) users will get invalidation notifiers on shareability changesI am sure there is a lot of work here and many quirks to be addressed, let's discuss this more with better context around. A few related RFC series are planned to be posted in the near future.
Look forward to collecting more context, and thanks for your patience while we catch up. But why not an iterative approach? We can't save struct page memory on guestmemfd huge pages until we have guestmemfd huge pages.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 14 lines · Rick P
Quoted 11 lines · Vishal Annapurve
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if the shareability changes, the users will get notified and will have to invalidate their mappings.guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.
Quoted 3 lines
I see the point about how operating on PFNs can allow smoother transition to a solution that saves struct page memory, but I wonder about the wisdom of building this 2MB TDX code against eventual goals.
This discussion was more in response to a few questions from Yan [1].
My point of this discussion was to ensure that:
1) There is more awareness about the future roadmap.
2) There is a line of sight towards supporting guest memory (at least guest private memory) without page structs.
No need to solve these problems right away, but it would be good to ensure that the design choices are aligned towards the future direction.
One thing that needs to be resolved right away is - no refcounts on guest memory from outside guest_memfd [2]. (Discounting the error situations)
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://lore.kernel.org/lkml/CAGtprH_ggm8N-R9QbV1f8mo8-cQkqyEta3W=h2jry-NRD7_6OA@mail.gmail.com/
Quoted 7 lines
We were thinking to enable 2MB TDX huge pages on top of:
1. Mmap shared pages
2. In-place conversion
3. 2MB huge page supportWhere do you think struct page-less guestmemfd fits in that roadmap?
Ideally the roadmap should be:
1. mmap support
2. Huge page support in guest memfd with in-place conversion
3. 2MB huge page EPT mappings support
4. private memory without page structs
5. private/shared memory without page structs
There should be newer RFC series landing soon for 1 and 2. In my opinion, as long as hugepage EPT support is reviewed, tested and is stable enough, it can land upstream sooner than 2 as well.
Quoted 16 lines
Quoted 9 lines
As an intermediate stage, it makes sense to me to just not have private memory backed by page structs and use a special "filemap" to map file offsets to these private memory ranges. This step will also need similar contract with users -
1) memory is pinned byguest_memfd
2) users will get invalidation notifiers on shareability changesQuoted 3 lines
I am sure there is a lot of work here and many quirks to be addressed, let's discuss this more with better context around. A few related RFC series are planned to be posted in the near future.
Look forward to collecting more context, and thanks for your patience while we catch up. But why not an iterative approach? We can't save struct page memory on guestmemfd huge pages until we have guestmemfd huge pages.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 6 lines · Vishal Annapurve
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
I see the point about how operating on PFNs can allow smoother transition to > > a
solution that saves struct page memory, but I wonder about the wisdom of building this 2MB TDX code against eventual goals.This discussion was more in response to a few questions from Yan [1].
Right, I follow.
Quoted 9 lines
Quoted 9 lines
My point of this discussion was to ensure that:
1) There is more awareness about the future roadmap.
2) There is a line of sight towards supporting guest memory (at least guest private memory) without page structs.No need to solve these problems right away, but it would be good to ensure that the design choices are aligned towards the future direction.
I'm not sure how much we should consider it at this stage. The kernel is not set in stone, so it's about how much you want to do at once. For us who have been working on the giant TDX base series, doing things on a more incremental smaller size sounds nice :). That said, the necessary changes may have other good reasons, as discussed.
Quoted 4 lines
Quoted 4 lines
One thing that needs to be resolved right away is - no refcounts on guest memory from outside
guest_memfd[2]. (Discounting the error situations)
Sounds fine.
Quoted 4 lines
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Vishal AnnapurveQuoted 88 lines · Vishal Annapurve
Quoted 79 lines · Yan Zhao
Quoted 70 lines · Vishal Annapurve
Quoted 69 lines · Yan Zhao
Quoted 67 lines · Vishal Annapurve
Quoted 51 lines · Yan Zhao
Quoted 44 lines · Vishal Annapurve
Quoted 43 lines · Yan Zhao
Quoted 41 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Sorry for the late reply, I was on leave last week.
Quoted 4 lines · Vishal Annapurve
Quoted 2 lines · Yan Zhao
So, we plan to remove folio_ref_add()/folio_put_refs() in future, only invoking
folio_ref_add()in the event of a removal failure.Quoted 8 lines
Quoted 2 lines
In my opinion, the above scheme can be deployed with this series itself.
guest_memfdwill not take away memory from TDX VMs without anI initially intended to add a separate patch at the end of this series to implement invoking
folio_ref_add()only upon a removal failure. However, I decided against it since it's not a must beforeguest_memfdsupports in-place conversion.We can include it in the next version If you think it's better.
Ackerley is planning to send out a series for 1G Hugetlb support with guest memfd soon, hopefully this week. Plus I don't see any reason to hold extra refcounts in TDX stack so it would be good to clean up this logic.
Quoted 7 lines
Quoted 3 lines
invalidation.
folio_ref_add()will not work for memory not backed by page structs, but that problem can be solved in future possibly byWith current TDX code, all memory must be backed by a page struct. Both
tdh_mem_page_add()andtdh_mem_page_aug()require a "struct page *" rather than a pfn.Quoted 6 lines
Quoted 2 lines
notifying
guest_memfdof certain ranges being in use even after invalidation completes.A curious question:
To support memory not backed by page structs in future, is there any counterpart to the page struct to hold ref count and map count?I imagine the needed support will match similar semantics as
VM_PFNMAP
[1] memory. No need to maintain refcounts/map counts for such physical memory ranges as all users will be notified when mappings are changed/removed.Quoted 5 lines
Quoted 2 lines
So, it's possible to map such memory in both shared and private EPT simultaneously?
No,
guest_memfdwill still ensure that userspace can only fault in shared memory regions in order to support CoCo VM usecases.Quoted 7 lines
Before
guest_memfdconverts a PFN from shared to private, how does it ensure there are no shared mappings? e.g., in [1], it uses the folio reference count to ensure that.Or do you believe that by eliminating the struct page, there would be no GUP, thereby ensuring no shared mappings by requiring all mappers to unmap in response to a
guest_memfdinvalidation notification?Yes.
Quoted 4 lines
As in
Documentation/core-api/pin_user_pages.rst, long-term pinning users have no need to register mmu notifier. So why users like VFIO must registerguest_memfdinvalidation notification?Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
VM_PFNMAP'd memory can't be long term pinned, so users of such memory ranges will have to adopt mechanisms to get notified. I think it would
Hmm, in current VFIO, it does not register any notifier for VM_PFNMAP'd memory.
I don't completely understand how VM_PFNMAP'd memory is used today for VFIO. Maybe only MMIO regions are backed by pfnmap today and the story for normal memory backed by pfnmap is yet to materialize.
Quoted 7 lines
VFIO can fault in VM_PFNMAP'd memory which is not from MMIO regions. It works because it knows VM_PFNMAP'd memory are always pinned.
Another example is udmabuf (
drivers/dma-buf/udmabuf.c), it mmaps normal folios withVM_PFNMAPflag without registering mmu notifier because those folios are pinned.
Quoted 7 lines
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if the
OK. So from point of the reset part of kernel, those pfns are not regarded as memory.
Quoted 4 lines
shareability changes, the users will get notified and will have to invalidate their mappings.
guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.
My concern is a failable invalidation notifer may not be ideal. Instead of relying on ref counts (or other mechanisms) to determine whether to start shareabilitiy changes, with a failable invalidation notifier, some users may fail the invalidation and the shareability change, even after other users have successfully unmapped a range.
Auditing whether multiple users of shared memory correctly perform unmapping is harder than auditing reference counts.
Quoted 9 lines
private memory backed by page structs and use a special "filemap" to map file offsets to these private memory ranges. This step will also need similar contract with users -
1) memory is pinned byguest_memfd
2) users will get invalidation notifiers on shareability changesI am sure there is a lot of work here and many quirks to be addressed, let's discuss this more with better context around. A few related RFC series are planned to be posted in the near future.
Ok. Thanks for your time and discussions :)
Quoted 123 lines
Quoted 42 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
be easy to pursue new users of
guest_memfdto follow this scheme. Irrespective of whether VM_PFNMAP'd support lands,guest_memfdhugepage support already needs the stance of: "Guest memfd owns all long-term refcounts on private memory" as discussed at LPC [1].[1] https://lpc.events/event/18/contributions/1764/attachments/1409/3182/LPC%202024_%201G%20page%20support%20for%20guest_memfd.pdf
(slide 12)Quoted 4 lines
Besides, how would
guest_memfdhandle potential unmap failures? e.g. what happens to prevent converting a private PFN to shared if there are errors when TDX unmaps a private PFN or if a device refuses to stop DMAing to a PFN.Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Users will have to signal such failures via the invalidation callback results or other appropriate mechanisms.
guest_memfdcan relay the failures up the call chain to the userspace.AFAIK, operations that perform actual unmapping do not allow failure, e.g.
kvm_mmu_unmap_gfn_range(),iopt_area_unfill_domains(),vfio_iommu_unmap_unpin_all(),vfio_iommu_unmap_unpin_reaccount().Very likely because these operations simply don't fail.
I think they are intentionally designed to be no-fail.
e.g. in
__iopt_area_unfill_domain(), no-fail is achieved by using a small backup buffer allocated on stack in case ofkmalloc()failure.Quoted 6 lines
Quoted 3 lines
That's why we rely on increasing folio ref count to reflect failure, which are due to unexpected SEAMCALL errors.
TDX stack is adding a scenario where invalidation can fail, a cleaner solution would be to propagate the result as an invalidation failure.
Not sure if linux kernel accepts unmap failure.
Quoted 2 lines
Another option is to notify
guest_memfdout of band to convey the ranges that failed invalidation.Yes, this might be better. Something similar like holding folio ref count to let
guest_memfdknow that a certain PFN cannot be re-assigned.Quoted 7 lines
Quoted 3 lines
With in-place conversion supported, even if the refcount is raised for such pages, they can still get used by the host if the
guest_memfdis unaware that the invalidation failed.I thought
guest_memfdshould check if folio ref count is 0 (or a base count) before conversion, splitting or re-assignment. Otherwise, why do you care if TDX holds the ref count? :)Soon to be posted RFC series by Ackerley currently explicitly checks for safe private page refcounts when folio splitting is needed and not for every private to shared conversion. A simple solution would be for
guest_memfdto check safe page refcounts for each private to shared conversion even if split is not required but will need to be reworked when either of the stages discussed above land where page structs are not around.Quoted 64 lines
Quoted 64 lines
Quoted 64 lines
Quoted 33 lines
Quoted 33 lines
Currently,
guest_memfdcan rely on page ref count to avoid re-assigning a PFN that fails to be unmapped.[1] https://lore.kernel.org/all/[email protected]/
Quoted 24 lines
Quoted 11 lines
Quoted 7 lines
Any
guest_memfdrange updates will result in invalidations/updates of userspace, guest, IOMMU or any other page tables referring toguest_memfdbacked pfns. This story will become clearer once the support for PFN range allocator for backingguest_memfdstarts getting discussed.Ok. It is indeed unclear right now to support such kind of memory.
Up to now, we don't anticipate TDX will allow any mapping of
VM_PFNMAPmemory into private EPT until TDX connect.There is a plan to use
VM_PFNMAPmemory for all ofguest_memfdshared/private ranges orthogonal to TDX connect usecase. With TDX connect/Sev TIO, major difference would be thatguest_memfdprivate ranges will be mapped into IOMMU page tables.Irrespective of whether/when
VM_PFNMAPmemory support lands, there have been discussions on not using page structs for private memory ranges altogether [1] even with hugetlb allocator, which will simplify seamless merge/split story for private hugepages to support memory conversion. So I think the general direction we should head towards is not relying on refcounts forguest_memfdprivate ranges and/or page structs altogether.Quoted 8 lines
Quoted 3 lines
It's fine to use PFN, but I wonder if there're counterparts of struct page to keep all necessary info.
Story will become clearer once VM_PFNMAP'd memory support starts getting discussed. In case of
guest_memfd, there is flexibility to store metadata for physical ranges withinguest_memfdjust like shareability tracking.Ok.
Quoted 21 lines
Quoted 20 lines
Quoted 4 lines
I think the series [2] to work better with PFNMAP'd physical memory in KVM is in the very right direction of not assuming page struct backed memory ranges for
guest_memfdas well.Note: Currently,
VM_PFNMAPis usually used together with flagVM_IO. in KVMhva_to_pfn_remapped()only applies to "vma->vm_flags& (VM_IO|VM_PFNMAP)".Quoted 12 lines
[1] https://lore.kernel.org/all/CAGtprH8akKUF=8+RkX_QMjp35C0bU1zxGi4v1Zm5AWCw=8V8AQ@mail.gmail.com/
[2] https://lore.kernel.org/linux-arm-kernel/[email protected]/Quoted 2 lines
And even in that scenario, the memory is only for private MMIO, so the backend driver is VFIO pci driver rather than
guest_memfd.Not necessary. As I mentioned above
guest_memfdranges will be backed byVM_PFNMAPmemory.Quoted 3 lines
Quoted 3 lines
[1] https://elixir.bootlin.com/linux/v6.14.5/source/mm/memory.c#L6543
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 16 lines · Yan Zhao
...
Quoted 8 lines
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if theOK. So from point of the reset part of kernel, those pfns are not regarded as memory.
Quoted 4 lines
shareability changes, the users will get notified and will have to invalidate their mappings.
guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.
Quoted 5 lines
My concern is a failable invalidation notifer may not be ideal. Instead of relying on ref counts (or other mechanisms) to determine whether to start shareabilitiy changes, with a failable invalidation notifier, some users may fail the invalidation and the shareability change, even after other users have successfully unmapped a range.
Even if one user fails to invalidate its mappings, I don't see a reason to go ahead with shareability change. Shareability should not change unless all existing users let go of their soon-to-be-invalid view of memory.
Quoted 15 lines
Auditing whether multiple users of shared memory correctly perform unmapping is harder than auditing reference counts.
Quoted 9 lines
private memory backed by page structs and use a special "filemap" to map file offsets to these private memory ranges. This step will also need similar contract with users -
1) memory is pinned byguest_memfd
2) users will get invalidation notifiers on shareability changesI am sure there is a lot of work here and many quirks to be addressed, let's discuss this more with better context around. A few related RFC series are planned to be posted in the near future.
Ok. Thanks for your time and discussions :)
...
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 23 lines · Vishal Annapurve
Quoted 16 lines · Yan Zhao
...
Quoted 8 lines
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if theOK. So from point of the reset part of kernel, those pfns are not regarded as memory.
Quoted 4 lines
shareability changes, the users will get notified and will have to invalidate their mappings.
guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.Quoted 5 lines
My concern is a failable invalidation notifer may not be ideal. Instead of relying on ref counts (or other mechanisms) to determine whether to start shareabilitiy changes, with a failable invalidation notifier, some users may fail the invalidation and the shareability change, even after other users have successfully unmapped a range.
Quoted 4 lines
Even if one user fails to invalidate its mappings, I don't see a reason to go ahead with shareability change. Shareability should not change unless all existing users let go of their soon-to-be-invalid view of memory.
My thinking is that:
1. guest_memfd starts shared-to-private conversion
2. guest_memfd sends invalidation notifications
2.1 invalidate notification --> A --> Unmap and return success 2.2 invalidate notification --> B --> Unmap and return success 2.3 invalidate notification --> C --> return failure
3. guest_memfd finds 2.3 fails, fails shared-to-private conversion and keeps shareability as shared
Though the GFN remains shared after 3, it's unmapped in user A and B in 2.1 and 2.2. Even if additional notifications could be sent to A and B to ask for mapping the GFN back, the map operation might fail. Consequently, A and B might not be able to restore the mapped status of the GFN. For IOMMU mappings, this could result in DMAR failure following a failed attempt to do shared-to-private conversion.
I noticed Ackerley has posted the series. Will check there later.
Quoted 15 lines
Quoted 15 lines
Auditing whether multiple users of shared memory correctly perform unmapping is harder than auditing reference counts.
Quoted 9 lines
private memory backed by page structs and use a special "filemap" to map file offsets to these private memory ranges. This step will also need similar contract with users -
1) memory is pinned byguest_memfd
2) users will get invalidation notifiers on shareability changesI am sure there is a lot of work here and many quirks to be addressed, let's discuss this more with better context around. A few related RFC series are planned to be posted in the near future.
Ok. Thanks for your time and discussions :)
...
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 24 lines · Yan Zhao
Quoted 23 lines · Vishal Annapurve
Quoted 21 lines · Yan Zhao
...
Quoted 8 lines
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if theOK. So from point of the reset part of kernel, those pfns are not regarded as memory.
Quoted 4 lines
shareability changes, the users will get notified and will have to invalidate their mappings.
guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.My concern is a failable invalidation notifer may not be ideal. Instead of relying on ref counts (or other mechanisms) to determine whether to start shareabilitiy changes, with a failable invalidation notifier, some users may fail the invalidation and the shareability change, even after other users have successfully unmapped a range.
Quoted 4 lines
Quoted 4 lines
Even if one user fails to invalidate its mappings, I don't see a reason to go ahead with shareability change. Shareability should not change unless all existing users let go of their soon-to-be-invalid view of memory.
Hi Yan,
While working on the 1G (aka HugeTLB) page support for guest_memfd series [1], we took into account conversion failures too. The steps are in kvm_gmem_convert_range(). (It might be easier to pull the entire series from GitHub [2] because the steps for conversion changed in two separate patches.)
We do need to handle errors across ranges to be converted, possibly from different memslots. The goal is to either have the entire conversion happen (including page split/merge) or nothing at all when the ioctl returns.
We try to undo the restructuring (whether split or merge) and undo any shareability changes on error (barring ENOMEM, in which case we leave a WARNing).
The part we don't restore is the presence of the pages in the host or guest page tables. For that, our idea is that if unmapped, the next access will just map it in, so there's no issue there.
Quoted 10 lines
My thinking is that:
1.
guest_memfdstarts shared-to-private conversion
2.guest_memfdsends invalidation notifications
2.1 invalidate notification --> A --> Unmap and return success 2.2 invalidate notification --> B --> Unmap and return success 2.3 invalidate notification --> C --> return failure
3.guest_memfdfinds 2.3 fails, fails shared-to-private conversion and keeps shareability as shared
Quoted 4 lines
Though the GFN remains shared after 3, it's unmapped in user A and B in 2.1 and 2.2. Even if additional notifications could be sent to A and B to ask for mapping the GFN back, the map operation might fail. Consequently, A and B might not be able to restore the mapped status of the GFN.
For conversion we don't attempt to restore mappings anywhere (whether in guest or host page tables). What do you think of not restoring the mappings?
Quoted 3 lines
For IOMMU mappings, this
could result in DMAR failure following a failed attempt to do shared-to-private conversion.
I believe the current conversion setup guards against this because after unmapping from the host, we check for any unexpected refcounts.
(This unmapping is not the unmapping we're concerned about, since this is shared memory, and unmapping doesn't go through TDX.)
Coming back to the refcounts, if the IOMMU had mappings, these refcounts are "unexpected". The conversion ioctl will return to userspace with an error.
IO can continue to happen, since the memory is still mapped in the IOMMU. The memory state is still shared. No issue there.
In RFCv2 [1], we expect userspace to see the error, then try and remove the memory from the IOMMU, and then try conversion again.
The part in concern here is unmapping failures of private pages, for private-to-shared conversions, since that part goes through TDX and might fail.
One other thing about taking refcounts is that in RFCv2, private-to-shared conversions assume that there are no refcounts on the private pages at all. (See filemap_remove_folio_for_restructuring() in [3])
Haven't had a chance to think about all the edge cases, but for now I think on unmapping failure, in addition to taking a refcount, we should return an error at least up to guest_memfd, so that guest_memfd could perhaps keep the refcount on that page, but drop the page from the filemap. Another option could be to track messed up addresses and always check that on conversion or something - not sure yet.
Either way, guest_memfd must know. If guest_memfd is not informed, on a next conversion request, the conversion will just spin in filemap_remove_folio_for_restructuring().
What do you think of this part about informing guest_memfd of the failure to unmap?
Quoted 3 lines
I noticed Ackerley has posted the series. Will check there later.
[1] https://lore.kernel.org/all/[email protected]/T/
[2] https://github.com/googleprodkernel/linux-cc/tree/gmem-1g-page-support-rfc-v2
[3] https://lore.kernel.org/all/7753dc66229663fecea2498cf442a768cb7191ba.1747264138.git.ackerleytng@google.com/
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Auditing whether multiple users of shared memory correctly perform unmapping is harder than auditing reference counts.
Quoted 9 lines
private memory backed by page structs and use a special "filemap" to map file offsets to these private memory ranges. This step will also need similar contract with users -
1) memory is pinned byguest_memfd
2) users will get invalidation notifiers on shareability changesI am sure there is a lot of work here and many quirks to be addressed, let's discuss this more with better context around. A few related RFC series are planned to be posted in the near future.
Ok. Thanks for your time and discussions :)
...
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 39 lines · Ackerley Tng
Quoted 24 lines · Yan Zhao
Quoted 23 lines · Vishal Annapurve
Quoted 21 lines · Yan Zhao
...
Quoted 8 lines
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if theOK. So from point of the reset part of kernel, those pfns are not regarded as memory.
Quoted 4 lines
shareability changes, the users will get notified and will have to invalidate their mappings.
guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.My concern is a failable invalidation notifer may not be ideal. Instead of relying on ref counts (or other mechanisms) to determine whether to start shareabilitiy changes, with a failable invalidation notifier, some users may fail the invalidation and the shareability change, even after other users have successfully unmapped a range.
Quoted 4 lines
Quoted 4 lines
Even if one user fails to invalidate its mappings, I don't see a reason to go ahead with shareability change. Shareability should not change unless all existing users let go of their soon-to-be-invalid view of memory.
Hi Yan,
While working on the 1G (aka HugeTLB) page support for
guest_memfdseries [1], we took into account conversion failures too. The steps are inkvm_gmem_convert_range(). (It might be easier to pull the entire series from GitHub [2] because the steps for conversion changed in two separate patches.)
Quoted 8 lines
We do need to handle errors across ranges to be converted, possibly from different memslots. The goal is to either have the entire conversion happen (including page split/merge) or nothing at all when the ioctl returns.
We try to undo the restructuring (whether split or merge) and undo any shareability changes on error (barring ENOMEM, in which case we leave a WARNing).
As the undo can fail (as the case you leave a WARNing, in patch 38 in [1]), it can lead to WARNings in kernel with folios not being properly added to the filemap.
Quoted 3 lines
The part we don't restore is the presence of the pages in the host or guest page tables. For that, our idea is that if unmapped, the next access will just map it in, so there's no issue there.
I don't think so.
As in patch 38 in [1], on failure, it may fail to
- restore the shareability
- restore the folio's filemap status
- restore the folio's hugetlb stash metadata
- restore the folio's merged/split status
Also, the host page table is not restored.
Quoted 10 lines
Quoted 10 lines
My thinking is that:
1.
guest_memfdstarts shared-to-private conversion
2.guest_memfdsends invalidation notifications
2.1 invalidate notification --> A --> Unmap and return success 2.2 invalidate notification --> B --> Unmap and return success 2.3 invalidate notification --> C --> return failure
3.guest_memfdfinds 2.3 fails, fails shared-to-private conversion and keeps shareability as shared
Quoted 8 lines
Quoted 4 lines
Though the GFN remains shared after 3, it's unmapped in user A and B in 2.1 and 2.2. Even if additional notifications could be sent to A and B to ask for mapping the GFN back, the map operation might fail. Consequently, A and B might not be able to restore the mapped status of the GFN.
For conversion we don't attempt to restore mappings anywhere (whether in guest or host page tables). What do you think of not restoring the mappings?
It could cause problem if the mappings in S-EPT can't be restored.
For TDX private-to-shared conversion, if kvm_gmem_convert_should_proceed() --> kvm_gmem_unmap_private() --> kvm_mmu_unmap_gfn_range() fails in the end, then the GFN shareability is restored to private. The next guest access to the partially unmapped private memory can meet a fatal error: "access before acceptance".
It could occur in such a scenario:
1. TD issues a TDVMCALL_MAP_GPA to convert a private GFN to shared
2. Conversion fails in KVM.
3. set_memory_decrypted() fails in TD.
4. TD thinks the GFN is still accepted as private and accesses it.
Quoted 6 lines
Quoted 3 lines
For IOMMU mappings, this
could result in DMAR failure following a failed attempt to do shared-to-private conversion.I believe the current conversion setup guards against this because after unmapping from the host, we check for any unexpected refcounts.
Right, it's fine if we check for any unexpected refcounts.
Quoted 7 lines
(This unmapping is not the unmapping we're concerned about, since this is shared memory, and unmapping doesn't go through TDX.)
Coming back to the refcounts, if the IOMMU had mappings, these refcounts are "unexpected". The conversion ioctl will return to userspace with an error.
Quoted 5 lines
IO can continue to happen, since the memory is still mapped in the IOMMU. The memory state is still shared. No issue there.
In RFCv2 [1], we expect userspace to see the error, then try and remove the memory from the IOMMU, and then try conversion again.
I don't think it's right to depend on that userspace could always perform in kernel's expected way, i.e. trying conversion until it succeeds.
We need to restore to the previous status (which includes the host page table) if conversion can't be done.
That said, in my view, a better flow would be:
1. guest_memfd sends a pre-invalidation request to users (users here means the consumers in kernel of memory allocated from guest_memfd).
2. Users (A, B, ..., X) perform pre-checks to determine if invalidation can proceed. For example, in the case of TDX, this might involve memory allocation and page splitting.
3. Based on the pre-check results, guest_memfd either aborts the invalidation or proceeds by sending the actual invalidation request.
4. Users (A-X) perform the actual unmap operation, ensuring it cannot fail. For TDX, the unmap must succeed unless there are bugs in the KVM or TDX module. In such cases, TDX can callback guest_memfd to inform the poison-status of the page or elevate the page reference count.
5. guest_memfd completes the invalidation process. If the memory is marked as "poison," guest_memfd can handle it accordingly. If the page has an elevated reference count, guest_memfd may not need to take special action, as the elevated count prevents the OS from reallocating the page.
(but from your reply below, seems a callback to guest_memfd is a better approach).
Quoted 3 lines
The part in concern here is unmapping failures of private pages, for private-to-shared conversions, since that part goes through TDX and might fail.
IMO, even for TDX, the real unmap must not fail unless there are bugs in the KVM or TDX module.
So, for page splitting in S-EPT, I prefer to try splitting in the pre-invalidation phase before conducting any real unmap.
Quoted 11 lines
One other thing about taking refcounts is that in RFCv2, private-to-shared conversions assume that there are no refcounts on the private pages at all. (See
filemap_remove_folio_for_restructuring()in [3])Haven't had a chance to think about all the edge cases, but for now I think on unmapping failure, in addition to taking a refcount, we should return an error at least up to
guest_memfd, so thatguest_memfdcould perhaps keep the refcount on that page, but drop the page from the filemap. Another option could be to track messed up addresses and always check that on conversion or something - not sure yet.
It looks good to me. See the bullet 4 in my proposed flow above.
Quoted 3 lines
Either way,
guest_memfdmust know. Ifguest_memfdis not informed, on a next conversion request, the conversion will just spin infilemap_remove_folio_for_restructuring().
It makes sense.
Quoted 2 lines
What do you think of this part about informing
guest_memfdof the failure to unmap?
So, do you want to add a guest_memfd callback to achieve this purpose?
BTW, here's an analysis of why we can't let kvm_mmu_unmap_gfn_range() and mmu_notifier_invalidate_range_start() fail, based on the repo
https://github.com/torvalds/linux.git, commit cd2e103d57e5 ("Merge tag 'hardening-v6.16-rc1-fix1-take2' ofgit://git.kernel.org/pub/scm/linux/kernel/git/kees/linux")
1. Status of mmu notifier
-------------------------------
(1) There're 34 direct callers of mmu_notifier_invalidate_range_start().
1. clear_refs_write
2. do_pagemap_scan
3. uprobe_write_opcode
4. do_huge_zero_wp_pmd
5. __split_huge_pmd (N)
6. __split_huge_pud (N)
7. move_pages_huge_pmd
8. copy_hugetlb_page_range
9. hugetlb_unshare_pmds (N)
10. hugetlb_change_protection
11. hugetlb_wp
12. unmap_hugepage_range (N)
13. move_hugetlb_page_tables
14. collapse_huge_page
15. retract_page_tables
16. collapse_pte_mapped_thp
17. write_protect_page
18. replace_page
19. madvise_free_single_vma
20. wp_clean_pre_vma
21. wp_page_copy
22. zap_page_range_single_batched (N)
23. unmap_vmas (N)
24. copy_page_range
25. remove_device_exclusive_entry
26. migrate_vma_collect
27. __migrate_device_pages
28. change_pud_range
29. move_page_tables
30. page_vma_mkclean_one
31. try_to_unmap_one
32. try_to_migrate_one
33. make_device_exclusive
34. move_pages_pte
Of these 34 direct callers, those marked with (N) cannot tolerate mmu_notifier_invalidate_range_start() failing. I have not yet investigated all 34 direct callers one by one, so the list of (N) is incomplete.
For 5. __split_huge_pmd(), Documentation/mm/transhuge.rst says:
"Note that split_huge_pmd() doesn't have any limitations on refcounting:
pmd can be split at any point and never fails." This is because split_huge_pmd() serves as a graceful fallback design for code walking pagetables but unaware about huge pmds.
(2) There's 1 direct caller of mmu_notifier_invalidate_range_start_nonblock(), __oom_reap_task_mm(), which only expects the error -EAGAIN.
In mn_hlist_invalidate_range_start():
"WARN_ON(mmu_notifier_range_blockable(range) || _ret != -EAGAIN);"
(3) For DMAs, drivers need to invoke pin_user_pages() to pin memory. In that case, they don't need to register mmu notifier.
Or, device drivers can pin pages via get_user_pages*(), and register for mmu notifier callbacks for the memory range. Then, upon receiving a notifier "invalidate range" callback , stop the device from using the range, and unpin the pages.
See Documentation/core-api/pin_user_pages.rst.
2. Cases that cannot tolerate failure of mmu_notifier_invalidate_range_start()
-------------------------------
(1) Error fallback cases.
1. split_huge_pmd() as mentioned in Documentation/mm/transhuge.rst.
split_huge_pmd() is designed as a graceful fallback without failure.
split_huge_pmd
|->__split_huge_pmd
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| split_huge_pmd_locked
| mmu_notifier_invalidate_range_end
2. in fs/iomap/buffered-io.c, iomap_write_failed() itself is error handling.
iomap_write_failed
|->truncate_pagecache_range
|->unmap_mapping_range
| |->unmap_mapping_pages
| |->unmap_mapping_range_tree
| |->unmap_mapping_range_vma
| |->zap_page_range_single
| |->zap_page_range_single_batched
| |->mmu_notifier_range_init
| | mmu_notifier_invalidate_range_start
| | unmap_single_vma
| | mmu_notifier_invalidate_range_end
|->truncate_inode_pages_range
|->truncate_cleanup_folio
|->if (folio_mapped(folio))
| unmap_mapping_folio(folio);
|->unmap_mapping_range_tree
|->unmap_mapping_range_vma
|->zap_page_range_single
|->zap_page_range_single_batched
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| unmap_single_vma
| mmu_notifier_invalidate_range_end 3. in mm/memory.c, zap_page_range_single() is invoked to handle error.
remap_pfn_range_notrack
|->int error = remap_pfn_range_internal(vma, addr, pfn, size, prot);
| if (!error)
| return 0;
| zap_page_range_single
|->zap_page_range_single_batched
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| unmap_single_vma
| mmu_notifier_invalidate_range_end 4. in kernel/events/core.c, zap_page_range_single() is invoked to clear any
partial mappings on error.
perf_mmap
|->ret = map_range(rb, vma);
| err = remap_pfn_range
|->if (err)
| zap_page_range_single
|->zap_page_range_single_batched
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| unmap_single_vma
| mmu_notifier_invalidate_range_end 5. in mm/memory.c, unmap_mapping_folio() is invoked to unmap posion page.
__do_fault
|->if (unlikely(PageHWPoison(vmf->page))) {
| vm_fault_t poisonret = VM_FAULT_HWPOISON;
| if (ret & VM_FAULT_LOCKED) {
| if (page_mapped(vmf->page))
| unmap_mapping_folio(folio);
| |->unmap_mapping_range_tree
| |->unmap_mapping_range_vma
| |->zap_page_range_single
| |->zap_page_range_single_batched
| |->mmu_notifier_range_init
| | mmu_notifier_invalidate_range_start
| | unmap_single_vma
| | mmu_notifier_invalidate_range_end
| if (mapping_evict_folio(folio->mapping, folio))
| poisonret = VM_FAULT_NOPAGE;
| folio_unlock(folio);
| }
| folio_put(folio);
| vmf->page = NULL;
| return poisonret;
| } 6. in mm/vma.c, in __mmap_region(), unmap_region() is invoked to undo any
partial mapping done by a device driver.
__mmap_new_vma
|->__mmap_new_file_vma(map, vma);
|->error = mmap_file(vma->vm_file, vma);
| if (error)
| unmap_region
|->unmap_vmas
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| unmap_single_vma
| mmu_notifier_invalidate_range_end(2) No-fail cases
-------------------------------
1. iput() cannot fail.
iput
|->iput_final
|->WRITE_ONCE(inode->i_state, state | I_FREEING);
| inode_lru_list_del(inode);
| evict(inode);
|->op->evict_inode(inode);
|->shmem_evict_inode
|->shmem_truncate_range
|->truncate_inode_pages_range
|->truncate_cleanup_folio
|->if (folio_mapped(folio))
| unmap_mapping_folio(folio);
|->unmap_mapping_range_tree
|->unmap_mapping_range_vma
|->zap_page_range_single
|->zap_page_range_single_batched
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| unmap_single_vma
| mmu_notifier_invalidate_range_end2. exit_mmap() cannot fail
exit_mmap
|->mmu_notifier_release(mm);
|->unmap_vmas(&tlb, &vmi.mas, vma, 0, ULONG_MAX, ULONG_MAX, false);
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| unmap_single_vma
| mmu_notifier_invalidate_range_end3. KVM Cases That Cannot Tolerate Unmap Failure
-------------------------------
Allowing unmap operations to fail in the following scenarios would make it very difficult or even impossible to handle the failure:
(1) __kvm_mmu_get_shadow_page() is designed to reliably obtain a shadow page without expecting any failure.
mmu_alloc_direct_roots
|->mmu_alloc_root
|->kvm_mmu_get_shadow_page
|->__kvm_mmu_get_shadow_page
|->kvm_mmu_alloc_shadow_page
|->account_shadowed
|->kvm_mmu_slot_gfn_write_protect
|->kvm_tdp_mmu_write_protect_gfn
|->write_protect_gfn
|->tdp_mmu_iter_set_spte(2) kvm_vfio_release() and kvm_vfio_file_del() cannot fail
kvm_vfio_release/kvm_vfio_file_del
|->kvm_vfio_update_coherency
|->kvm_arch_unregister_noncoherent_dma
|->kvm_noncoherent_dma_assignment_start_or_stop
|->kvm_zap_gfn_range
|->kvm_tdp_mmu_zap_leafs
|->tdp_mmu_zap_leafs
|->tdp_mmu_iter_set_spte(3) There're lots of callers of __kvm_set_or_clear_apicv_inhibit() currently never expect failure of unmap.
__kvm_set_or_clear_apicv_inhibit
|->kvm_zap_gfn_range
|->kvm_tdp_mmu_zap_leafs
|->tdp_mmu_zap_leafs
|->tdp_mmu_iter_set_spte4. Cases in KVM where it's hard to make tdp_mmu_set_spte() (update SPTE with write mmu_lock) failable.
(1) kvm_vcpu_flush_tlb_guest()
kvm_vcpu_flush_tlb_guest
|->kvm_mmu_sync_roots
|->mmu_sync_children
|->kvm_vcpu_write_protect_gfn
|->kvm_mmu_slot_gfn_write_protect
|->kvm_tdp_mmu_write_protect_gfn
|->write_protect_gfn
|->tdp_mmu_iter_set_spte
|->tdp_mmu_set_spte(2) handle_removed_pt() and handle_changed_spte().
Thanks
Yan
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 45 lines · Yan Zhao
Quoted 44 lines · Ackerley Tng
Quoted 24 lines · Yan Zhao
Quoted 23 lines · Vishal Annapurve
Quoted 21 lines · Yan Zhao
...
Quoted 8 lines
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if theOK. So from point of the reset part of kernel, those pfns are not regarded as memory.
Quoted 4 lines
shareability changes, the users will get notified and will have to invalidate their mappings.
guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.My concern is a failable invalidation notifer may not be ideal. Instead of relying on ref counts (or other mechanisms) to determine whether to start shareabilitiy changes, with a failable invalidation notifier, some users may fail the invalidation and the shareability change, even after other users have successfully unmapped a range.
Quoted 4 lines
Quoted 4 lines
Even if one user fails to invalidate its mappings, I don't see a reason to go ahead with shareability change. Shareability should not change unless all existing users let go of their soon-to-be-invalid view of memory.
Hi Yan,
While working on the 1G (aka HugeTLB) page support for
guest_memfdseries [1], we took into account conversion failures too. The steps are inkvm_gmem_convert_range(). (It might be easier to pull the entire series from GitHub [2] because the steps for conversion changed in two separate patches.)We do need to handle errors across ranges to be converted, possibly from different memslots. The goal is to either have the entire conversion happen (including page split/merge) or nothing at all when the ioctl returns.
Quoted 7 lines
Quoted 3 lines
We try to undo the restructuring (whether split or merge) and undo any shareability changes on error (barring ENOMEM, in which case we leave a WARNing).
As the undo can fail (as the case you leave a WARNing, in patch 38 in [1]), it can lead to WARNings in kernel with folios not being properly added to the filemap.
I'm not sure how else to handle errors on rollback path. I've hopefully addressed this on the other thread at [1].
Quoted 4 lines
Quoted 3 lines
The part we don't restore is the presence of the pages in the host or guest page tables. For that, our idea is that if unmapped, the next access will just map it in, so there's no issue there.
Quoted 8 lines
I don't think so.
As in patch 38 in [1], on failure, it may fail to
- restore the shareability
- restore the folio's filemap status
- restore the folio's hugetlb stash metadata
- restore the folio's merged/split status
The plan is that we try our best to restore shareability, filemap, restructuring (aka split/merge, including stash metadata) other than failures on rollback.
Quoted 3 lines
Also, the host page table is not restored.
This is by design, the host page tables can be re-populated on the next fault. I've hopefully addressed this on the other thread at [1].
Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
My thinking is that:
1.
guest_memfdstarts shared-to-private conversion
2.guest_memfdsends invalidation notifications
2.1 invalidate notification --> A --> Unmap and return success 2.2 invalidate notification --> B --> Unmap and return success 2.3 invalidate notification --> C --> return failure
3.guest_memfdfinds 2.3 fails, fails shared-to-private conversion and keeps shareability as sharedQuoted 8 lines
Quoted 4 lines
Though the GFN remains shared after 3, it's unmapped in user A and B in 2.1 and 2.2. Even if additional notifications could be sent to A and B to ask for mapping the GFN back, the map operation might fail. Consequently, A and B might not be able to restore the mapped status of the GFN.
For conversion we don't attempt to restore mappings anywhere (whether in guest or host page tables). What do you think of not restoring the mappings?
It could cause problem if the mappings in S-EPT can't be restored.
For TDX private-to-shared conversion, if
kvm_gmem_convert_should_proceed()-->kvm_gmem_unmap_private()-->kvm_mmu_unmap_gfn_range()fails in the end, then the GFN shareability is restored to private. The next guest access to the partially unmapped private memory can meet a fatal error: "access before acceptance".
Quoted 7 lines
It could occur in such a scenario:
1. TD issues aTDVMCALL_MAP_GPAto convert a private GFN to shared
2. Conversion fails in KVM.
3.set_memory_decrypted()fails in TD.
4. TD thinks the GFN is still accepted as private and accesses it.
This is true, I was thinking that this isn't handled solely in conversion but by being part of the contract between userspace VMM and the guest, that guest must handle conversion failures. I've hopefully addressed this on the other thread at [1].
Quoted 16 lines
Quoted 6 lines
Quoted 3 lines
For IOMMU mappings, this
could result in DMAR failure following a failed attempt to do shared-to-private conversion.I believe the current conversion setup guards against this because after unmapping from the host, we check for any unexpected refcounts.
Right, it's fine if we check for any unexpected refcounts.
Quoted 7 lines
(This unmapping is not the unmapping we're concerned about, since this is shared memory, and unmapping doesn't go through TDX.)
Coming back to the refcounts, if the IOMMU had mappings, these refcounts are "unexpected". The conversion ioctl will return to userspace with an error.
Quoted 8 lines
Quoted 5 lines
IO can continue to happen, since the memory is still mapped in the IOMMU. The memory state is still shared. No issue there.
In RFCv2 [1], we expect userspace to see the error, then try and remove the memory from the IOMMU, and then try conversion again.
I don't think it's right to depend on that userspace could always perform in kernel's expected way, i.e. trying conversion until it succeeds.
Let me think more deeply about this. Please let me know if there's anything I missed.
It is true that a buggy or malicious userspace VMM can ignore conversion failures and report success to the guest, but if both the userspace VMM and guest are malicious, it's quite hard for the kernel to defend against that.
I think as long as there's no point where the guest can crash the host in a fixed way, I think it is okay to rely on a userspace VMM and guest protocol.
IIUC the guest can crash the host (original point of having guest_memfd) if the guest can convince the host to write to private memory. For that to happen, the memory must be faulted into the Secure EPTs, and the shareability state must be ALL for the host to fault it in.
So to have this issue, the conversion failure must be such that the memory remains faulted into the Secure EPTs while shareability is shared. Since unmapping from secure EPTs happens pretty early before any shareability is changed or any rollback (and rollback failures) can happen, I think we should be quite safe?
If unmapping of private memory fails, this is where I think guest_memfd should get an error from the unmap and it should not proceed to change shareability.
Quoted 2 lines
We need to restore to the previous status (which includes the host page table) if conversion can't be done.
Most of the previous status (shareability, filemap, restructuring (aka split/merge, including stash metadata)) are restored other than during rollback failures.
As for presence in host page tables, is it okay to defer that till the next fault, and if not okay, why not?
For presence in guest page tables, is it okay to fall back on the protocol where the guest must handle conversion failures, and if not okay, why not?
Quoted 17 lines
That said, in my view, a better flow would be:
1.
guest_memfdsends a pre-invalidation request to users (users here means the consumers in kernel of memory allocated fromguest_memfd).2. Users (A, B, ..., X) perform pre-checks to determine if invalidation can proceed. For example, in the case of TDX, this might involve memory allocation and page splitting.
3. Based on the pre-check results,
guest_memfdeither aborts the invalidation or proceeds by sending the actual invalidation request.4. Users (A-X) perform the actual unmap operation, ensuring it cannot fail. For TDX, the unmap must succeed unless there are bugs in the KVM or TDX module. In such cases, TDX can callback
guest_memfdto inform the poison-status of the page or elevate the page reference count.
Quoted 8 lines
5.
guest_memfdcompletes the invalidation process. If the memory is marked as "poison,"guest_memfdcan handle it accordingly. If the page has an elevated reference count,guest_memfdmay not need to take special action, as the elevated count prevents the OS from reallocating the page.
(but from your reply below, seems a callback toguest_memfdis a better approach).
Thanks for this, I've tried to combine this into my response at [1]. I think this works, but it's hard because
a. Pre-checks are hard to check (explained at [1])
b. Even after all the checks, unmapping can still fail, and those still
have to be handled, and to handle those, we have to buy into the userspace VMM/guest protocol, so why not just buy into the protocol to start with?
[1] https://lore.kernel.org/all/[email protected]/
Quoted 9 lines
Quoted 3 lines
The part in concern here is unmapping failures of private pages, for private-to-shared conversions, since that part goes through TDX and might fail.
IMO, even for TDX, the real unmap must not fail unless there are bugs in the KVM or TDX module.
So, for page splitting in S-EPT, I prefer to try splitting in the pre-invalidation phase before conducting any real unmap.
Thanks for your detailed suggestion.
Quoted 5 lines
Quoted 5 lines
One other thing about taking refcounts is that in RFCv2, private-to-shared conversions assume that there are no refcounts on the private pages at all. (See
filemap_remove_folio_for_restructuring()in [3])
Quoted 9 lines
Quoted 6 lines
Haven't had a chance to think about all the edge cases, but for now I think on unmapping failure, in addition to taking a refcount, we should return an error at least up to
guest_memfd, so thatguest_memfdcould perhaps keep the refcount on that page, but drop the page from the filemap. Another option could be to track messed up addresses and always check that on conversion or something - not sure yet.It looks good to me. See the bullet 4 in my proposed flow above.
Thanks again for your detailed suggestion.
Quoted 10 lines
Quoted 3 lines
Either way,
guest_memfdmust know. Ifguest_memfdis not informed, on a next conversion request, the conversion will just spin infilemap_remove_folio_for_restructuring().It makes sense.
Quoted 2 lines
What do you think of this part about informing
guest_memfdof the failure to unmap?So, do you want to add a
guest_memfdcallback to achieve this purpose?
I will need to think the entire thing through, but I meant informing as in returning an error to guest_memfd so that guest_memfd knows. I think returning an error should be the first cause of action.
As for whether guest_memfd should know how to handle the error or whether the userspace VMM should participate in deciding what to do with the error, I'm not sure. If you have suggestions on this, I hope we can combine the suggestions about the conversion protocol on the other thread.
Regarding a callback, are you thinking something like not having the unmap return an error, but instead TDX will call a function like kvm_gmem_error_at_offset(loff_t offset), and guest_memfd will then record that somewhere, and then immediately after calling unmap guest_memfd will check kvm_gmem_was_there_an_error_in_range() and then determining whether there's an error? Something like that?
I guess it could work but feels a little odd.
Quoted 6 lines
BTW, here's an analysis of why we can't let
kvm_mmu_unmap_gfn_range()andmmu_notifier_invalidate_range_start()fail, based on the repo
https://github.com/torvalds/linux.git, commitcd2e103d57e5("Merge tag 'hardening-v6.16-rc1-fix1-take2' ofgit://git.kernel.org/pub/scm/linux/kernel/git/kees/linux")
Thank you, I appreciate the effort you took to enumerate these. The following suggestions are based on my current understanding. I don't have time in the near future to do the plumbing to test out the suggestion, but for now I want to see if this suggestion makes sense, maybe you can correct any misunderstandings first.
Quoted 43 lines
1. Status of mmu notifier
-------------------------------
(1) There're 34 direct callers ofmmu_notifier_invalidate_range_start().
1.clear_refs_write
2.do_pagemap_scan
3.uprobe_write_opcode
4.do_huge_zero_wp_pmd
5.__split_huge_pmd(N)
6.__split_huge_pud(N)
7.move_pages_huge_pmd
8.copy_hugetlb_page_range
9.hugetlb_unshare_pmds(N)
10.hugetlb_change_protection
11.hugetlb_wp
12.unmap_hugepage_range(N)
13.move_hugetlb_page_tables
14.collapse_huge_page
15.retract_page_tables
16.collapse_pte_mapped_thp
17.write_protect_page
18.replace_page
19.madvise_free_single_vma
20.wp_clean_pre_vma
21.wp_page_copy
22.zap_page_range_single_batched(N)
23.unmap_vmas(N)
24.copy_page_range
25.remove_device_exclusive_entry
26.migrate_vma_collect
27.__migrate_device_pages
28.change_pud_range
29.move_page_tables
30.page_vma_mkclean_one
31.try_to_unmap_one
32.try_to_migrate_one
33.make_device_exclusive
34.move_pages_pteOf these 34 direct callers, those marked with (N) cannot tolerate
mmu_notifier_invalidate_range_start()failing. I have not yet investigated all 34 direct callers one by one, so the list of (N) is incomplete.
Quoted 7 lines
For 5.
__split_huge_pmd(),Documentation/mm/transhuge.rstsays:
"Note thatsplit_huge_pmd()doesn't have any limitations on refcounting:
pmd can be split at any point and never fails." This is becausesplit_huge_pmd()serves as a graceful fallback design for code walking pagetables but unaware about huge pmds.
Do these callers, especially those with (N), ever try to unmap any TDX private pages? guest_memfd only gives shared pages to core-mm, so for shared pages, there will continue to be no chance of errors.
If we change mmu_notifier_invalidate_range_start() to return an int, all of the callers that never invalidate shared pages can continue to safely rely on the fact that mmu_notifier_invalidate_range_start() will return 0.
For the callers of mmu_notifier_invalidate_range_start() that may touch private pages, I believe that's only guest_memfd and KVM. That's where we want the error, and will handle the error.
Another point here is that I was thinking to put EPT splitting together with actual unmapping instead of with invalidation because we will probably invalidate more than we unmap (see explanation at [1] about the race). Maybe moving EPT splitting to unmap could help?
Quoted 10 lines
(2) There's 1 direct caller of
mmu_notifier_invalidate_range_start_nonblock(),__oom_reap_task_mm(), which only expects the error-EAGAIN.In
mn_hlist_invalidate_range_start():
"WARN_ON(mmu_notifier_range_blockable(range) || _ret != -EAGAIN);"(3) For DMAs, drivers need to invoke
pin_user_pages()to pin memory. In that case, they don't need to register mmu notifier.
Quoted 8 lines
Or, device drivers can pin pages via get_user_pages*(), and register for mmu notifier callbacks for the memory range. Then, upon receiving a notifier "invalidate range" callback , stop the device from using the range, and unpin the pages.
See
Documentation/core-api/pin_user_pages.rst.
Do you mean that we should teach device drivers to get callbacks for private pages? Are you looking ahead to handle TDX IO on private pages? So far we haven't handled that yet.
Quoted 95 lines
2. Cases that cannot tolerate failure of
mmu_notifier_invalidate_range_start()
-------------------------------
(1) Error fallback cases.1. split_huge_pmd() as mentioned in Documentation/mm/transhuge.rst. split_huge_pmd() is designed as a graceful fallback without failure. split_huge_pmd |->__split_huge_pmd |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | split_huge_pmd_locked | mmu_notifier_invalidate_range_end 2. in fs/iomap/buffered-io.c, iomap_write_failed() itself is error handling. iomap_write_failed |->truncate_pagecache_range |->unmap_mapping_range | |->unmap_mapping_pages | |->unmap_mapping_range_tree | |->unmap_mapping_range_vma | |->zap_page_range_single | |->zap_page_range_single_batched | |->mmu_notifier_range_init | | mmu_notifier_invalidate_range_start | | unmap_single_vma | | mmu_notifier_invalidate_range_end |->truncate_inode_pages_range |->truncate_cleanup_folio |->if (folio_mapped(folio)) | unmap_mapping_folio(folio); |->unmap_mapping_range_tree |->unmap_mapping_range_vma |->zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end3. in mm/memory.c, zap_page_range_single() is invoked to handle error. remap_pfn_range_notrack |->int error = remap_pfn_range_internal(vma, addr, pfn, size, prot); | if (!error) | return 0; | zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end 4. in kernel/events/core.c, zap_page_range_single() is invoked to clear any partial mappings on error. perf_mmap |->ret = map_range(rb, vma); | err = remap_pfn_range |->if (err) | zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end 5. in mm/memory.c, unmap_mapping_folio() is invoked to unmap posion page. __do_fault |->if (unlikely(PageHWPoison(vmf->page))) { | vm_fault_t poisonret = VM_FAULT_HWPOISON; | if (ret & VM_FAULT_LOCKED) { | if (page_mapped(vmf->page)) | unmap_mapping_folio(folio); | |->unmap_mapping_range_tree | |->unmap_mapping_range_vma | |->zap_page_range_single | |->zap_page_range_single_batched | |->mmu_notifier_range_init | | mmu_notifier_invalidate_range_start | | unmap_single_vma | | mmu_notifier_invalidate_range_end | if (mapping_evict_folio(folio->mapping, folio)) | poisonret = VM_FAULT_NOPAGE; | folio_unlock(folio); | } | folio_put(folio); | vmf->page = NULL; | return poisonret; | }6. in
mm/vma.c, in__mmap_region(),unmap_region()is invoked to undo any
Quoted 14 lines
partial mapping done by a device driver.
__mmap_new_vma
|->__mmap_new_file_vma(map, vma);
|->error = mmap_file(vma->vm_file, vma);
| if (error)
| unmap_region
|->unmap_vmas
|->mmu_notifier_range_init
| mmu_notifier_invalidate_range_start
| unmap_single_vma
| mmu_notifier_invalidate_range_endThese should probably not ever be invalidating or unmapping private pages.
Quoted 28 lines
(2) No-fail cases
-------------------------------
1.iput()cannot fail.iput
|->iput_final|->WRITE_ONCE(inode->i_state, state | I_FREEING); | inode_lru_list_del(inode); | evict(inode); |->op->evict_inode(inode); |->shmem_evict_inode |->shmem_truncate_range |->truncate_inode_pages_range |->truncate_cleanup_folio |->if (folio_mapped(folio)) | unmap_mapping_folio(folio); |->unmap_mapping_range_tree |->unmap_mapping_range_vma |->zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end2.
exit_mmap()cannot fail
Quoted 9 lines
exit_mmap|->mmu_notifier_release(mm); |->unmap_vmas(&tlb, &vmi.mas, vma, 0, ULONG_MAX, ULONG_MAX, false); |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end
These should probably not ever be invalidating or unmapping private pages.
Quoted 10 lines
3. KVM Cases That Cannot Tolerate Unmap Failure
-------------------------------
Allowing unmap operations to fail in the following scenarios would make it very difficult or even impossible to handle the failure:(1)
__kvm_mmu_get_shadow_page()is designed to reliably obtain a shadow page without expecting any failure.
mmu_alloc_direct_roots
|->mmu_alloc_root
Quoted 10 lines
|->kvm_mmu_get_shadow_page
|->__kvm_mmu_get_shadow_page
|->kvm_mmu_alloc_shadow_page
|->account_shadowed
|->kvm_mmu_slot_gfn_write_protect
|->kvm_tdp_mmu_write_protect_gfn
|->write_protect_gfn
|->tdp_mmu_iter_set_spteI need to learn more about shadow pages but IIUC TDX doesn't use shadow pages so this path won't interact with unmapping private pages.
Quoted 12 lines
(2)
kvm_vfio_release()andkvm_vfio_file_del()cannot failkvm_vfio_release/kvm_vfio_file_del
|->kvm_vfio_update_coherency|->kvm_arch_unregister_noncoherent_dma |->kvm_noncoherent_dma_assignment_start_or_stop |->kvm_zap_gfn_range |->kvm_tdp_mmu_zap_leafs |->tdp_mmu_zap_leafs |->tdp_mmu_iter_set_spte
I need to learn more about VFIO but for now IIUC IO uses shared pages, so this path won't interact with unmapping private pages.
Quoted 11 lines
(3) There're lots of callers of
__kvm_set_or_clear_apicv_inhibit()currently never expect failure of unmap.
__kvm_set_or_clear_apicv_inhibit
|->kvm_zap_gfn_range|->kvm_tdp_mmu_zap_leafs |->tdp_mmu_zap_leafs |->tdp_mmu_iter_set_spte
There could be some TDX specific things such that TDX doesn't use this path.
Quoted 16 lines
4. Cases in KVM where it's hard to make
tdp_mmu_set_spte()(update SPTE with writemmu_lock) failable.(1)
kvm_vcpu_flush_tlb_guest()
kvm_vcpu_flush_tlb_guest
|->kvm_mmu_sync_roots|->mmu_sync_children |->kvm_vcpu_write_protect_gfn |->kvm_mmu_slot_gfn_write_protect |->kvm_tdp_mmu_write_protect_gfn |->write_protect_gfn |->tdp_mmu_iter_set_spte |->tdp_mmu_set_spte
Quoted 2 lines
(2)
handle_removed_pt()andhandle_changed_spte().
Thank you so much for looking into these, I'm hoping that the number of cases where TDX and private pages are unmapped are really limited to a few paths that we have to rework.
If we agree that the error has to be handled, then regardless of how we let the caller know that an error happened, all paths touching TDX private pages have to be reworked.
Between (1) returning an error vs (2) marking error and having the caller check for errors, then it's probably better to use the standard approach of returning an error since it is better understood, and there's no need to have extra data structures?
Quoted 3 lines
Thanks
Yan
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 50 lines · Ackerley Tng
Quoted 48 lines · Yan Zhao
Quoted 47 lines · Ackerley Tng
Quoted 24 lines · Yan Zhao
Quoted 23 lines · Vishal Annapurve
Quoted 21 lines · Yan Zhao
...
Quoted 8 lines
I might be wrongly throwing out some terminologies here then.
VM_PFNMAPflag can be set for memory backed by folios/page structs. udmabuf seems to be working with pinned "folios" in the backend.The goal is to get to a stage where
guest_memfdis backed by pfn ranges unmanaged by kernel thatguest_memfdowns and distributes to userspace, KVM, IOMMU subject to shareability attributes. if theOK. So from point of the reset part of kernel, those pfns are not regarded as memory.
Quoted 4 lines
shareability changes, the users will get notified and will have to invalidate their mappings.
guest_memfdwill allow mmaping such ranges withVM_PFNMAPflag set by default in the VMAs to indicate the need of special handling/lack of page structs.My concern is a failable invalidation notifer may not be ideal. Instead of relying on ref counts (or other mechanisms) to determine whether to start shareabilitiy changes, with a failable invalidation notifier, some users may fail the invalidation and the shareability change, even after other users have successfully unmapped a range.
Quoted 4 lines
Quoted 4 lines
Even if one user fails to invalidate its mappings, I don't see a reason to go ahead with shareability change. Shareability should not change unless all existing users let go of their soon-to-be-invalid view of memory.
Hi Yan,
While working on the 1G (aka HugeTLB) page support for
guest_memfdseries [1], we took into account conversion failures too. The steps are inkvm_gmem_convert_range(). (It might be easier to pull the entire series from GitHub [2] because the steps for conversion changed in two separate patches.)We do need to handle errors across ranges to be converted, possibly from different memslots. The goal is to either have the entire conversion happen (including page split/merge) or nothing at all when the ioctl returns.
We try to undo the restructuring (whether split or merge) and undo any shareability changes on error (barring ENOMEM, in which case we leave a WARNing).
Quoted 7 lines
Quoted 4 lines
As the undo can fail (as the case you leave a WARNing, in patch 38 in [1]), it can lead to WARNings in kernel with folios not being properly added to the filemap.
I'm not sure how else to handle errors on rollback path. I've hopefully addressed this on the other thread at [1].
I'll reply [1].
Please also check my reply and proposal at [2].
Quoted 17 lines
Quoted 4 lines
Quoted 3 lines
The part we don't restore is the presence of the pages in the host or guest page tables. For that, our idea is that if unmapped, the next access will just map it in, so there's no issue there.
Quoted 8 lines
I don't think so.
As in patch 38 in [1], on failure, it may fail to
- restore the shareability
- restore the folio's filemap status
- restore the folio's hugetlb stash metadata
- restore the folio's merged/split statusThe plan is that we try our best to restore shareability, filemap, restructuring (aka split/merge, including stash metadata) other than failures on rollback.
Quoted 6 lines
Quoted 3 lines
Also, the host page table is not restored.
This is by design, the host page tables can be re-populated on the next fault. I've hopefully addressed this on the other thread at [1].
This is not. Please check my reply to [1].
Quoted 64 lines
Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
My thinking is that:
1.
guest_memfdstarts shared-to-private conversion
2.guest_memfdsends invalidation notifications
2.1 invalidate notification --> A --> Unmap and return success 2.2 invalidate notification --> B --> Unmap and return success 2.3 invalidate notification --> C --> return failure
3.guest_memfdfinds 2.3 fails, fails shared-to-private conversion and keeps shareability as sharedQuoted 8 lines
Quoted 4 lines
Though the GFN remains shared after 3, it's unmapped in user A and B in 2.1 and 2.2. Even if additional notifications could be sent to A and B to ask for mapping the GFN back, the map operation might fail. Consequently, A and B might not be able to restore the mapped status of the GFN.
For conversion we don't attempt to restore mappings anywhere (whether in guest or host page tables). What do you think of not restoring the mappings?
It could cause problem if the mappings in S-EPT can't be restored.
For TDX private-to-shared conversion, if
kvm_gmem_convert_should_proceed()-->kvm_gmem_unmap_private()-->kvm_mmu_unmap_gfn_range()fails in the end, then the GFN shareability is restored to private. The next guest access to the partially unmapped private memory can meet a fatal error: "access before acceptance".Quoted 7 lines
It could occur in such a scenario:
1. TD issues aTDVMCALL_MAP_GPAto convert a private GFN to shared
2. Conversion fails in KVM.
3.set_memory_decrypted()fails in TD.
4. TD thinks the GFN is still accepted as private and accesses it.This is true, I was thinking that this isn't handled solely in conversion but by being part of the contract between userspace VMM and the guest, that guest must handle conversion failures. I've hopefully addressed this on the other thread at [1].
Quoted 16 lines
Quoted 6 lines
Quoted 3 lines
For IOMMU mappings, this
could result in DMAR failure following a failed attempt to do shared-to-private conversion.I believe the current conversion setup guards against this because after unmapping from the host, we check for any unexpected refcounts.
Right, it's fine if we check for any unexpected refcounts.
Quoted 7 lines
(This unmapping is not the unmapping we're concerned about, since this is shared memory, and unmapping doesn't go through TDX.)
Coming back to the refcounts, if the IOMMU had mappings, these refcounts are "unexpected". The conversion ioctl will return to userspace with an error.
Quoted 8 lines
Quoted 5 lines
IO can continue to happen, since the memory is still mapped in the IOMMU. The memory state is still shared. No issue there.
In RFCv2 [1], we expect userspace to see the error, then try and remove the memory from the IOMMU, and then try conversion again.
I don't think it's right to depend on that userspace could always perform in kernel's expected way, i.e. trying conversion until it succeeds.
Quoted 7 lines
Let me think more deeply about this. Please let me know if there's anything I missed.
It is true that a buggy or malicious userspace VMM can ignore conversion failures and report success to the guest, but if both the userspace VMM and guest are malicious, it's quite hard for the kernel to defend against that.
Hmm, expecting userspace to try conversion endlessly exceeds what is reasonable for a cooperative userspace?
Quoted 6 lines
I think as long as there's no point where the guest can crash the host in a fixed way, I think it is okay to rely on a userspace VMM and guest protocol.
IIUC the guest can crash the host (original point of having
guest_memfd) if the guest can convince the host to write to private memory. For that
How to?
Unless the host kernel wants to crash itself, I don't think allowing guest to crash the host is acceptable.
If you happen to know one, please let us know. We'll fix it.
Quoted 8 lines
to happen, the memory must be faulted into the Secure EPTs, and the shareability state must be ALL for the host to fault it in.
So to have this issue, the conversion failure must be such that the memory remains faulted into the Secure EPTs while shareability is shared. Since unmapping from secure EPTs happens pretty early before any shareability is changed or any rollback (and rollback failures) can happen, I think we should be quite safe?
It's not safe if unmapping from the secure EPT fails while the shareability is changed to shared.
Quoted 3 lines
If unmapping of private memory fails, this is where I think
guest_memfdshould get an error from the unmap and it should not proceed to change shareability.
Please check if my proposal at [2] is agreeable.
Quoted 7 lines
Quoted 3 lines
We need to restore to the previous status (which includes the host page table) if conversion can't be done.
Most of the previous status (shareability, filemap, restructuring (aka split/merge, including stash metadata)) are restored other than during rollback failures.
However, error during the rollback is unacceptable.
Quoted 2 lines
As for presence in host page tables, is it okay to defer that till the next fault, and if not okay, why not?
If the host page tables involve only shared mappings in the primary MMU and shared EPT, it's ok.
Quoted 3 lines
For presence in guest page tables, is it okay to fall back on the protocol where the guest must handle conversion failures, and if not okay, why not?
Hmm, whether to roll back the guest page table or not after the conversion failure is the business of the guest OS.
However, KVM can't rely on that the guest must assume that the page state is shared even after a private-to-shared conversion failure.
Quoted 26 lines
Quoted 17 lines
That said, in my view, a better flow would be:
1.
guest_memfdsends a pre-invalidation request to users (users here means the consumers in kernel of memory allocated fromguest_memfd).2. Users (A, B, ..., X) perform pre-checks to determine if invalidation can proceed. For example, in the case of TDX, this might involve memory allocation and page splitting.
3. Based on the pre-check results,
guest_memfdeither aborts the invalidation or proceeds by sending the actual invalidation request.4. Users (A-X) perform the actual unmap operation, ensuring it cannot fail. For TDX, the unmap must succeed unless there are bugs in the KVM or TDX module. In such cases, TDX can callback
guest_memfdto inform the poison-status of the page or elevate the page reference count.Quoted 8 lines
5.
guest_memfdcompletes the invalidation process. If the memory is marked as "poison,"guest_memfdcan handle it accordingly. If the page has an elevated reference count,guest_memfdmay not need to take special action, as the elevated count prevents the OS from reallocating the page.
(but from your reply below, seems a callback toguest_memfdis a better approach).
Quoted 4 lines
Thanks for this, I've tried to combine this into my response at [1]. I think this works, but it's hard because
a. Pre-checks are hard to check (explained at [1])
Please check if the pre-checks in my POC [2] is good.
I tested it for the case of TDX unmapping failure. It does not change the shareabilitiy if splitting or zapping fails.
Quoted 4 lines
b. Even after all the checks, unmapping can still fail, and those still
have to be handled, and to handle those, we have to buy into the userspace VMM/guest protocol, so why not just buy into the protocol to start with?
In my POC [2], the outcome of unmapping failure is to leak the pages. Please check if it looks good to you.
Quoted 1 line
[2] https://lore.kernel.org/all/aE%[email protected]/
Quoted 65 lines
Quoted 9 lines
Quoted 3 lines
The part in concern here is unmapping failures of private pages, for private-to-shared conversions, since that part goes through TDX and might fail.
IMO, even for TDX, the real unmap must not fail unless there are bugs in the KVM or TDX module.
So, for page splitting in S-EPT, I prefer to try splitting in the pre-invalidation phase before conducting any real unmap.Thanks for your detailed suggestion.
Quoted 5 lines
Quoted 5 lines
One other thing about taking refcounts is that in RFCv2, private-to-shared conversions assume that there are no refcounts on the private pages at all. (See
filemap_remove_folio_for_restructuring()in [3])Quoted 9 lines
Quoted 6 lines
Haven't had a chance to think about all the edge cases, but for now I think on unmapping failure, in addition to taking a refcount, we should return an error at least up to
guest_memfd, so thatguest_memfdcould perhaps keep the refcount on that page, but drop the page from the filemap. Another option could be to track messed up addresses and always check that on conversion or something - not sure yet.It looks good to me. See the bullet 4 in my proposed flow above.
Thanks again for your detailed suggestion.
Quoted 10 lines
Quoted 3 lines
Either way,
guest_memfdmust know. Ifguest_memfdis not informed, on a next conversion request, the conversion will just spin infilemap_remove_folio_for_restructuring().It makes sense.
Quoted 2 lines
What do you think of this part about informing
guest_memfdof the failure to unmap?So, do you want to add a
guest_memfdcallback to achieve this purpose?I will need to think the entire thing through, but I meant informing as in returning an error to
guest_memfdso thatguest_memfdknows. I think returning an error should be the first cause of action.As for whether
guest_memfdshould know how to handle the error or whether the userspace VMM should participate in deciding what to do with the error, I'm not sure. If you have suggestions on this, I hope we can combine the suggestions about the conversion protocol on the other thread.Regarding a callback, are you thinking something like not having the unmap return an error, but instead TDX will call a function like
kvm_gmem_error_at_offset(loff_t offset), andguest_memfdwill then record that somewhere, and then immediately after calling unmapguest_memfdwill checkkvm_gmem_was_there_an_error_in_range()and then determining whether there's an error? Something like that?I guess it could work but feels a little odd.
Quoted 6 lines
BTW, here's an analysis of why we can't let
kvm_mmu_unmap_gfn_range()andmmu_notifier_invalidate_range_start()fail, based on the repo
https://github.com/torvalds/linux.git, commitcd2e103d57e5("Merge tag 'hardening-v6.16-rc1-fix1-take2' ofgit://git.kernel.org/pub/scm/linux/kernel/git/kees/linux")
Quoted 5 lines
Thank you, I appreciate the effort you took to enumerate these. The following suggestions are based on my current understanding. I don't have time in the near future to do the plumbing to test out the suggestion, but for now I want to see if this suggestion makes sense, maybe you can correct any misunderstandings first.
Sorry, I realized that the below enumeration brought confusion.
Listing them was to prove that unmapping failure is not expected by the kernel.
Please kindly let me know if any existing kernel code allows unmap to fail.
Quoted 50 lines
Quoted 42 lines
1. Status of mmu notifier
-------------------------------
(1) There're 34 direct callers ofmmu_notifier_invalidate_range_start().
1.clear_refs_write
2.do_pagemap_scan
3.uprobe_write_opcode
4.do_huge_zero_wp_pmd
5.__split_huge_pmd(N)
6.__split_huge_pud(N)
7.move_pages_huge_pmd
8.copy_hugetlb_page_range
9.hugetlb_unshare_pmds(N)
10.hugetlb_change_protection
11.hugetlb_wp
12.unmap_hugepage_range(N)
13.move_hugetlb_page_tables
14.collapse_huge_page
15.retract_page_tables
16.collapse_pte_mapped_thp
17.write_protect_page
18.replace_page
19.madvise_free_single_vma
20.wp_clean_pre_vma
21.wp_page_copy
22.zap_page_range_single_batched(N)
23.unmap_vmas(N)
24.copy_page_range
25.remove_device_exclusive_entry
26.migrate_vma_collect
27.__migrate_device_pages
28.change_pud_range
29.move_page_tables
30.page_vma_mkclean_one
31.try_to_unmap_one
32.try_to_migrate_one
33.make_device_exclusive
34.move_pages_pteOf these 34 direct callers, those marked with (N) cannot tolerate
mmu_notifier_invalidate_range_start()failing. I have not yet investigated all 34 direct callers one by one, so the list of (N) is incomplete.Quoted 7 lines
For 5.
__split_huge_pmd(),Documentation/mm/transhuge.rstsays:
"Note thatsplit_huge_pmd()doesn't have any limitations on refcounting:
pmd can be split at any point and never fails." This is becausesplit_huge_pmd()serves as a graceful fallback design for code walking pagetables but unaware about huge pmds.
Quoted 8 lines
Do these callers, especially those with (N), ever try to unmap any TDX private pages?
guest_memfdonly gives shared pages to core-mm, so for shared pages, there will continue to be no chance of errors.If we change
mmu_notifier_invalidate_range_start()to return an int, all of the callers that never invalidate shared pages can continue to safely rely on the fact thatmmu_notifier_invalidate_range_start()will return 0.
mmu_notifier_invalidate_range_start() is just to zap shared pages.
Quoted 24 lines
For the callers of
mmu_notifier_invalidate_range_start()that may touch private pages, I believe that's onlyguest_memfdand KVM. That's where we want the error, and will handle the error.Another point here is that I was thinking to put EPT splitting together with actual unmapping instead of with invalidation because we will probably invalidate more than we unmap (see explanation at [1] about the race). Maybe moving EPT splitting to unmap could help?
Quoted 15 lines
(2) There's 1 direct caller of
mmu_notifier_invalidate_range_start_nonblock(),__oom_reap_task_mm(), which only expects the error-EAGAIN.In
mn_hlist_invalidate_range_start():
"WARN_ON(mmu_notifier_range_blockable(range) || _ret != -EAGAIN);"(3) For DMAs, drivers need to invoke
pin_user_pages()to pin memory. In that case, they don't need to register mmu notifier.Or, device drivers can pin pages via get_user_pages*(), and register for mmu notifier callbacks for the memory range. Then, upon receiving a notifier "invalidate range" callback , stop the device from using the range, and unpin the pages.
Quoted 7 lines
Quoted 3 lines
See
Documentation/core-api/pin_user_pages.rst.Do you mean that we should teach device drivers to get callbacks for private pages? Are you looking ahead to handle TDX IO on private pages? So far we haven't handled that yet.
I tried to show that device drivers increases page refcount (by pinning) when it maps a page into IOMMU page table. It does not decrease page refcount (by unpinning) until after unmapping.
If the page hold by the device driver is allocated from hugetlb, and if the page has been truncated from the hugetlb, the page is still hold by the device driver until the page is unmapped in the IOMMU page table.
This is similar to TDX. As long as a page is still mapped in the SEPT or tracked by the TDX module, it's better to hold a page refcount even after the page is truncated from the file mapping.
Quoted 204 lines
Quoted 95 lines
2. Cases that cannot tolerate failure of
mmu_notifier_invalidate_range_start()
-------------------------------
(1) Error fallback cases.1. split_huge_pmd() as mentioned in Documentation/mm/transhuge.rst. split_huge_pmd() is designed as a graceful fallback without failure. split_huge_pmd |->__split_huge_pmd |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | split_huge_pmd_locked | mmu_notifier_invalidate_range_end 2. in fs/iomap/buffered-io.c, iomap_write_failed() itself is error handling. iomap_write_failed |->truncate_pagecache_range |->unmap_mapping_range | |->unmap_mapping_pages | |->unmap_mapping_range_tree | |->unmap_mapping_range_vma | |->zap_page_range_single | |->zap_page_range_single_batched | |->mmu_notifier_range_init | | mmu_notifier_invalidate_range_start | | unmap_single_vma | | mmu_notifier_invalidate_range_end |->truncate_inode_pages_range |->truncate_cleanup_folio |->if (folio_mapped(folio)) | unmap_mapping_folio(folio); |->unmap_mapping_range_tree |->unmap_mapping_range_vma |->zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end3. in mm/memory.c, zap_page_range_single() is invoked to handle error. remap_pfn_range_notrack |->int error = remap_pfn_range_internal(vma, addr, pfn, size, prot); | if (!error) | return 0; | zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end 4. in kernel/events/core.c, zap_page_range_single() is invoked to clear any partial mappings on error. perf_mmap |->ret = map_range(rb, vma); | err = remap_pfn_range |->if (err) | zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end 5. in mm/memory.c, unmap_mapping_folio() is invoked to unmap posion page. __do_fault |->if (unlikely(PageHWPoison(vmf->page))) { | vm_fault_t poisonret = VM_FAULT_HWPOISON; | if (ret & VM_FAULT_LOCKED) { | if (page_mapped(vmf->page)) | unmap_mapping_folio(folio); | |->unmap_mapping_range_tree | |->unmap_mapping_range_vma | |->zap_page_range_single | |->zap_page_range_single_batched | |->mmu_notifier_range_init | | mmu_notifier_invalidate_range_start | | unmap_single_vma | | mmu_notifier_invalidate_range_end | if (mapping_evict_folio(folio->mapping, folio)) | poisonret = VM_FAULT_NOPAGE; | folio_unlock(folio); | } | folio_put(folio); | vmf->page = NULL; | return poisonret; | }6. in
mm/vma.c, in__mmap_region(),unmap_region()is invoked to undo anyQuoted 14 lines
partial mapping done by a device driver. __mmap_new_vma |->__mmap_new_file_vma(map, vma); |->error = mmap_file(vma->vm_file, vma); | if (error) | unmap_region |->unmap_vmas |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_endThese should probably not ever be invalidating or unmapping private pages.
Quoted 28 lines
(2) No-fail cases
-------------------------------
1.iput()cannot fail.iput
|->iput_final|->WRITE_ONCE(inode->i_state, state | I_FREEING); | inode_lru_list_del(inode); | evict(inode); |->op->evict_inode(inode); |->shmem_evict_inode |->shmem_truncate_range |->truncate_inode_pages_range |->truncate_cleanup_folio |->if (folio_mapped(folio)) | unmap_mapping_folio(folio); |->unmap_mapping_range_tree |->unmap_mapping_range_vma |->zap_page_range_single |->zap_page_range_single_batched |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_end2.
exit_mmap()cannot failQuoted 9 lines
exit_mmap|->mmu_notifier_release(mm); |->unmap_vmas(&tlb, &vmi.mas, vma, 0, ULONG_MAX, ULONG_MAX, false); |->mmu_notifier_range_init | mmu_notifier_invalidate_range_start | unmap_single_vma | mmu_notifier_invalidate_range_endThese should probably not ever be invalidating or unmapping private pages.
Quoted 10 lines
3. KVM Cases That Cannot Tolerate Unmap Failure
-------------------------------
Allowing unmap operations to fail in the following scenarios would make it very difficult or even impossible to handle the failure:(1)
__kvm_mmu_get_shadow_page()is designed to reliably obtain a shadow page without expecting any failure.
mmu_alloc_direct_roots
|->mmu_alloc_rootQuoted 10 lines
|->kvm_mmu_get_shadow_page |->__kvm_mmu_get_shadow_page |->kvm_mmu_alloc_shadow_page |->account_shadowed |->kvm_mmu_slot_gfn_write_protect |->kvm_tdp_mmu_write_protect_gfn |->write_protect_gfn |->tdp_mmu_iter_set_spteI need to learn more about shadow pages but IIUC TDX doesn't use shadow pages so this path won't interact with unmapping private pages.
Quoted 12 lines
(2)
kvm_vfio_release()andkvm_vfio_file_del()cannot failkvm_vfio_release/kvm_vfio_file_del
|->kvm_vfio_update_coherency|->kvm_arch_unregister_noncoherent_dma |->kvm_noncoherent_dma_assignment_start_or_stop |->kvm_zap_gfn_range |->kvm_tdp_mmu_zap_leafs |->tdp_mmu_zap_leafs |->tdp_mmu_iter_set_spteI need to learn more about VFIO but for now IIUC IO uses shared pages, so this path won't interact with unmapping private pages.
Quoted 11 lines
(3) There're lots of callers of
__kvm_set_or_clear_apicv_inhibit()currently never expect failure of unmap.
__kvm_set_or_clear_apicv_inhibit
|->kvm_zap_gfn_range|->kvm_tdp_mmu_zap_leafs |->tdp_mmu_zap_leafs |->tdp_mmu_iter_set_spte
Quoted 2 lines
There could be some TDX specific things such that TDX doesn't use this path.
tdp_mmu_iter_set_spte() is used by KVM generally to update the SPTE when kvm->mmu_lock is held for write.
TDX uses tdp_mmu_iter_set_spte() to further unmapping the SEPT.
Converting tdp_mmu_iter_set_spte() to return error is a huge work and I don't think it's right or worthwhile.
Quoted 24 lines
Quoted 17 lines
4. Cases in KVM where it's hard to make
tdp_mmu_set_spte()(update SPTE with writemmu_lock) failable.(1)
kvm_vcpu_flush_tlb_guest()
kvm_vcpu_flush_tlb_guest
|->kvm_mmu_sync_roots|->mmu_sync_children |->kvm_vcpu_write_protect_gfn |->kvm_mmu_slot_gfn_write_protect |->kvm_tdp_mmu_write_protect_gfn |->write_protect_gfn |->tdp_mmu_iter_set_spte |->tdp_mmu_set_spteQuoted 2 lines
(2)
handle_removed_pt()andhandle_changed_spte().Thank you so much for looking into these, I'm hoping that the number of cases where TDX and private pages are unmapped are really limited to a few paths that we have to rework.
Quoted 8 lines
If we agree that the error has to be handled, then regardless of how we let the caller know that an error happened, all paths touching TDX private pages have to be reworked.
Between (1) returning an error vs (2) marking error and having the caller check for errors, then it's probably better to use the standard approach of returning an error since it is better understood, and there's no need to have extra data structures?
However, I don't think returning error during the unmap path is a standard approach...
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 4 lines · Yan Zhao
Quoted 4 lines
It is true that a buggy or malicious userspace VMM can ignore conversion failures and report success to the guest, but if both the userspace VMM and guest are malicious, it's quite hard for the kernel to defend against that.
For upstream, it's going to be required that userspace can't mess up the host kernel. Userspace is free to mess up the guest though.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Yan ZhaoQuoted 12 lines · Yan Zhao
We need to restore to the previous status (which includes the host page table) if conversion can't be done.
That said, in my view, a better flow would be:1.
guest_memfdsends a pre-invalidation request to users (users here means the consumers in kernel of memory allocated fromguest_memfd).2. Users (A, B, ..., X) perform pre-checks to determine if invalidation can proceed. For example, in the case of TDX, this might involve memory allocation and page splitting.
Quoted 7 lines
3. Based on the pre-check results,
guest_memfdeither aborts the invalidation or proceeds by sending the actual invalidation request.4. Users (A-X) perform the actual unmap operation, ensuring it cannot fail. For TDX, the unmap must succeed unless there are bugs in the KVM or TDX module. In such cases, TDX can callback
guest_memfdto inform the poison-status of the page or elevate the page reference count.
Few questions here:
1) It sounds like the failure to remove entries from SEPT could only be due to bugs in the KVM/TDX module, how reliable would it be to continue executing TDX VMs on the host once such bugs are hit?
2) Is it reliable to continue executing the host kernel and other normal VMs once such bugs are hit?
3) Can the memory be reclaimed reliably if the VM is marked as dead and cleaned up right away?
Quoted 8 lines
5.
guest_memfdcompletes the invalidation process. If the memory is marked as "poison,"guest_memfdcan handle it accordingly. If the page has an elevated reference count,guest_memfdmay not need to take special action, as the elevated count prevents the OS from reallocating the page.
(but from your reply below, seems a callback toguest_memfdis a better approach).
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 16 lines · Vishal Annapurve
Quoted 15 lines · Yan Zhao
We need to restore to the previous status (which includes the host page table) if conversion can't be done.
That said, in my view, a better flow would be:1.
guest_memfdsends a pre-invalidation request to users (users here means the consumers in kernel of memory allocated fromguest_memfd).2. Users (A, B, ..., X) perform pre-checks to determine if invalidation can proceed. For example, in the case of TDX, this might involve memory allocation and page splitting.
3. Based on the pre-check results,
guest_memfdeither aborts the invalidation or proceeds by sending the actual invalidation request.
Quoted 8 lines
Quoted 4 lines
4. Users (A-X) perform the actual unmap operation, ensuring it cannot fail. For TDX, the unmap must succeed unless there are bugs in the KVM or TDX module. In such cases, TDX can callback
guest_memfdto inform the poison-status of the page or elevate the page reference count.Few questions here:
1) It sounds like the failure to remove entries from SEPT could only be due to bugs in the KVM/TDX module,
Yes.
Quoted 2 lines
how reliable would it be to
continue executing TDX VMs on the host once such bugs are hit?
The TDX VMs will be killed. However, the private pages are still mapped in the SEPT (after the unmapping failure).
The teardown flow for TDX VM is:
do_exit
|->exit_files
|->kvm_gmem_release ==> (1) Unmap guest pages
|->release kvmfd
|->kvm_destroy_vm (2) Reclaiming resources
|->kvm_arch_pre_destroy_vm ==> Release hkid
|->kvm_arch_destroy_vm ==> Reclaim SEPT page table pagesWithout holding page reference after (1) fails, the guest pages may have been re-assigned by the host OS while they are still still tracked in the TDX module.
Quoted 2 lines
2) Is it reliable to continue executing the host kernel and other normal VMs once such bugs are hit?
If with TDX holding the page ref count, the impact of unmapping failure of guest pages is just to leak those pages.
Quoted 2 lines
3) Can the memory be reclaimed reliably if the VM is marked as dead and cleaned up right away?
As in the above flow, TDX needs to hold the page reference on unmapping failure until after reclaiming is successful. Well, reclaiming itself is possible to fail either.
So, below is my proposal. Showed in the simple POC code based on
https://github.com/googleprodkernel/linux-cc/commits/wip-tdx-gmem-conversions-hugetlb-2mept-v2.
Patch 1: TDX increases page ref count on unmap failure. Patch 2: Bail out private-to-shared conversion if splitting fails. Patch 3: Make kvm_gmem_zap() return void.
After the change,
- the actual private-to-shared conversion will be not be executed on splitting failure (which could be due to out of memory or bugs in the KVM/TDX module) or unmapping failure (which is due to bugs in the KVM/TDX module).
- other callers of kvm_gmem_zap(), such as kvm_gmem_release(), kvm_gmem_error_folio(), kvm_gmem_punch_hole(), are still allowed to proceed. After truncating the pages out of the filemap, the pages could be leaked on purpose with the reference hold by TDX.
commit 50432c0bb1e10591714b6b880f43fc30797ca047
Author: Yan Zhao <[email protected]> Date: Tue Jun 10 00:02:30 2025 -0700
KVM: TDX: Hold folio ref count on fatal error
arch/x86/kvm/vmx/tdx.c +22 -4
@@ -1502,6 +1502,15 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)td_vmcs_write64 (to_tdx (vcpu),SHARED_EPT_POINTER , root_hpa); } +/* +* Called when fatal error occurs during removing a TD's page. +* Increase the folio ref count in case it's reused by other VMs or host. +*/ +static void tdx_hold_page_on_error (struct kvm *kvm,struct page *page,int level) +{ +folio_ref_add (page_folio (page),KVM_PAGES_PER_HPAGE (level)); +} +static int tdx_mem_page_aug (struct kvm *kvm,gfn_t gfn,enum pg_level level,struct page *page) {
@@ -1868,12 +1877,14 @@ int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,* before any might be populated. Warn if zapping is attempted when * there can't be anything populated in the private EPT. */ -if (KVM_BUG_ON (!is_hkid_assigned (to_kvm_tdx (kvm)), kvm)) -return -EINVAL ; +if (KVM_BUG_ON (!is_hkid_assigned (to_kvm_tdx (kvm)), kvm)) { + ret = -EINVAL ; + goto fatal_error; + } ret =tdx_sept_zap_private_spte (kvm, gfn, level, page);if (ret <=0 ) -return ret; + goto fatal_error;/* * TDX requires TLB tracking before dropping private page. Do
@@ -1881,7 +1892,14 @@ int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,*/ tdx_track (kvm); -return tdx_sept_drop_private_spte (kvm, gfn, level, page); + ret =tdx_sept_drop_private_spte (kvm, gfn, level, page); +if (ret) + goto fatal_error; +return ret; +fatal_error: +if (ret <0 ) +tdx_hold_page_on_error (kvm, page, level); +return ret; } void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode,
commit 240acb13d4bd724b4c153b73cfba3cd14d3cc296
Author: Yan Zhao <[email protected]> Date: Tue Jun 10 19:26:56 2025 -0700
KVM: guest_memfd: Add check kvm_gmem_private_has_safe_refcount()
Check extra ref count on private pages in case of TDX unmap failure before private to shared conversion in the backend.
In other zap cases, it's ok to do without this ref count check so that the error folio will be held by TDX after guest_memfd releases the folio.
virt/kvm/guest_memfd.c +39 -0
@@ -521,6 +521,41 @@ static void kvm_gmem_convert_invalidate_end(struct inode *inode,kvm_gmem_invalidate_end (gmem, invalidate_start, invalidate_end); } +static bool kvm_gmem_private_has_safe_refcount (struct inode *inode, +pgoff_t start,pgoff_t end) +{ +pgoff_t index = start; +size_t inode_nr_pages; +bool ret = true; +void *priv; + +/* +* Conversion in !kvm_gmem_has_custom_allocator() case does not reach here. +*/ +if (!kvm_gmem_has_custom_allocator (inode)) +return ret; + + priv =kvm_gmem_allocator_private (inode); + inode_nr_pages =kvm_gmem_allocator_ops (inode)->nr_pages_in_folio (priv); + +while (index < end) { +struct folio *f; + f =filemap_get_folio (inode->i_mapping, index); +if (IS_ERR (f)) { + index += inode_nr_pages; +continue ; + } + +folio_put (f); +if (folio_ref_count (f) >folio_nr_pages (f)) { + ret = false; +break ; + } + index +=folio_nr_pages (f); + } +return ret; +} +static int kvm_gmem_convert_should_proceed (struct inode *inode,struct conversion_work *work,bool to_shared,pgoff_t *error_index)
@@ -538,6 +573,10 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,list_for_each_entry (gmem, gmem_list, entry) { ret =kvm_gmem_zap (gmem, work->start, work_end,KVM_FILTER_PRIVATE , true); +if (ret) +return ret; +if (!kvm_gmem_private_has_safe_refcount (inode, work->start, work_end)) +return -EFAULT ; } }else { unmap_mapping_pages(inode->i_mapping, work->start,
commit 26743993663313fa6f8741a43f22ed5ac21399c7
Author: Yan Zhao <[email protected]> Date: Tue Jun 10 20:01:23 2025 -0700
KVM: guest_memfd: Move splitting KVM mappings out of kvm_gmem_zap()
Modify kvm_gmem_zap() to return void and introduce a separate function, kvm_gmem_split_private(), to handle the splitting of private EPT.
With these changes, kvm_gmem_zap() will only be executed after successful splitting across the entire conversion/punch range.
virt/kvm/guest_memfd.c +49 -16
119 lines not on this page · raw message
If the above changes are agreeable, we could consider a more ambitious approach:
introducing an interface like:
int guest_memfd_add_page_ref_count(gfn_t gfn, int nr); int guest_memfd_dec_page_ref_count(gfn_t gfn, int nr);
This would allow guest_memfd to maintain an internal reference count for each private GFN. TDX would call guest_memfd_add_page_ref_count() for mapping and guest_memfd_dec_page_ref_count() after a successful unmapping. Before truncating a private page from the filemap, guest_memfd could increase the real folio reference count based on its internal reference count for the private GFN.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 5 lines · Yan Zhao
If the above changes are agreeable, we could consider a more ambitious approach:
introducing an interface like:int
guest_memfd_add_page_ref_count(gfn_t gfn, int nr); intguest_memfd_dec_page_ref_count(gfn_t gfn, int nr);
We talked about doing something like having tdx_hold_page_on_error() in guestmemfd with a proper name. The separation of concerns will be better if we can just tell guestmemfd, the page has an issue. Then guestmemfd can decide how to handle it (refcount or whatever).
Quoted 6 lines
This would allow
guest_memfdto maintain an internal reference count for each private GFN. TDX would callguest_memfd_add_page_ref_count()for mapping andguest_memfd_dec_page_ref_count()after a successful unmapping. Before truncating a private page from the filemap,guest_memfdcould increase the real folio reference count based on its internal reference count for the private GFN.
What does this get us exactly? This is the argument to have less error prone code that can survive forgetting to refcount on error? I don't see that it is an especially special case.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 11 lines · Rick P
Quoted 5 lines · Yan Zhao
If the above changes are agreeable, we could consider a more ambitious approach:
introducing an interface like:int
guest_memfd_add_page_ref_count(gfn_t gfn, int nr); intguest_memfd_dec_page_ref_count(gfn_t gfn, int nr);We talked about doing something like having
tdx_hold_page_on_error()in guestmemfd with a proper name. The separation of concerns will be better if we can just tell guestmemfd, the page has an issue. Then guestmemfd can decide how to handle it (refcount or whatever).
Instead of using tdx_hold_page_on_error(), the advantage of informing guest_memfd that TDX is holding a page at 4KB granularity is that, even if there is a bug in KVM (such as forgetting to notify TDX to remove a mapping in handle_removed_pt()), guest_memfd would be aware that the page remains mapped in the TDX module. This allows guest_memfd to determine how to handle the problematic page (whether through refcount adjustments or other methods) before truncating it.
Quoted 10 lines
Quoted 6 lines
This would allow
guest_memfdto maintain an internal reference count for each private GFN. TDX would callguest_memfd_add_page_ref_count()for mapping andguest_memfd_dec_page_ref_count()after a successful unmapping. Before truncating a private page from the filemap,guest_memfdcould increase the real folio reference count based on its internal reference count for the private GFN.What does this get us exactly? This is the argument to have less error prone code that can survive forgetting to refcount on error? I don't see that it is an especially special case.
Yes, for a less error prone code.
If this approach is considered too complex for an initial implementation, using tdx_hold_page_on_error() is also a viable option.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 6 lines · Yan Zhao
Quoted 6 lines
We talked about doing something like having
tdx_hold_page_on_error()in guestmemfd with a proper name. The separation of concerns will be better if we
can just tell guestmemfd, the page has an issue. Then guestmemfd can decide how
to handle it (refcount or whatever).
Quoted 10 lines
Instead of using
tdx_hold_page_on_error(), the advantage of informingguest_memfdthat TDX is holding a page at 4KB granularity is that, even if there
is a bug in KVM (such as forgetting to notify TDX to remove a mapping inhandle_removed_pt()),guest_memfdwould be aware that the page remains mapped in
the TDX module. This allowsguest_memfdto determine how to handle the problematic page (whether through refcount adjustments or other methods) before
truncating it.
I don't think a potential bug in KVM is a good enough reason. If we are concerned can we think about a warning instead?
We had talked enhancing kasan to know when a page is mapped into S-EPT in the past. So rather than design around potential bugs we could focus on having a simpler implementation with the infrastructure to catch and fix the bugs.
Quoted 15 lines
Quoted 11 lines
Quoted 10 lines
This would allow
guest_memfdto maintain an internal reference count for each
private GFN. TDX would callguest_memfd_add_page_ref_count()for mapping andguest_memfd_dec_page_ref_count()after a successful unmapping. Before truncating
a private page from the filemap,guest_memfdcould increase the real folio reference count based on its internal reference count for the private GFN.Quoted 4 lines
What does this get us exactly? This is the argument to have less error prone code that can survive forgetting to refcount on error? I don't see that it is an
especially special case.
Quoted 5 lines
Yes, for a less error prone code.
If this approach is considered too complex for an initial implementation, using
tdx_hold_page_on_error()is also a viable option.
I'm saying I don't think it's not a good enough reason. Why is it different then other use-after free bugs? I feel like I'm missing something.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 18 lines · Rick P
Quoted 6 lines · Yan Zhao
Quoted 6 lines
We talked about doing something like having
tdx_hold_page_on_error()in guestmemfd with a proper name. The separation of concerns will be better if we
can just tell guestmemfd, the page has an issue. Then guestmemfd can decide how
to handle it (refcount or whatever).Quoted 10 lines
Instead of using
tdx_hold_page_on_error(), the advantage of informingguest_memfdthat TDX is holding a page at 4KB granularity is that, even if there
is a bug in KVM (such as forgetting to notify TDX to remove a mapping inhandle_removed_pt()),guest_memfdwould be aware that the page remains mapped in
the TDX module. This allowsguest_memfdto determine how to handle the problematic page (whether through refcount adjustments or other methods) before
truncating it.
Quoted 6 lines
I don't think a potential bug in KVM is a good enough reason. If we are concerned can we think about a warning instead?
We had talked enhancing kasan to know when a page is mapped into S-EPT in the past. So rather than design around potential bugs we could focus on having a simpler implementation with the infrastructure to catch and fix the bugs.
However, if failing to remove a guest private page would only cause memory leak, it's fine.
If TDX does not hold any refcount, guest_memfd has to know that which private page is still mapped. Otherwise, the page may be re-assigned to other kernel components while it may still be mapped in the S-EPT.
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
This would allow
guest_memfdto maintain an internal reference count for each
private GFN. TDX would callguest_memfd_add_page_ref_count()for mapping andguest_memfd_dec_page_ref_count()after a successful unmapping. Before truncating
a private page from the filemap,guest_memfdcould increase the real folio reference count based on its internal reference count for the private GFN.What does this get us exactly? This is the argument to have less error prone code that can survive forgetting to refcount on error? I don't see that it is an
especially special case.
Quoted 8 lines
Quoted 5 lines
Yes, for a less error prone code.
If this approach is considered too complex for an initial implementation, using
tdx_hold_page_on_error()is also a viable option.I'm saying I don't think it's not a good enough reason. Why is it different then other use-after free bugs? I feel like I'm missing something.
By tdx_hold_page_on_error(), it could be implememented as on removal failure, invoke a guest_memfd interface to let guest_memfd know exact ranges still being under use by the TDX module due to unmapping failures.
Do you think it's ok?
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 7 lines · Yan Zhao
Quoted 7 lines
I don't think a potential bug in KVM is a good enough reason. If we are concerned can we think about a warning instead?
We had talked enhancing kasan to know when a page is mapped into S-EPT in the
past. So rather than design around potential bugs we could focus on having a simpler implementation with the infrastructure to catch and fix the bugs.
Quoted 6 lines
However, if failing to remove a guest private page would only cause memory leak,
it's fine.
If TDX does not hold any refcount,guest_memfdhas to know that which private page is still mapped. Otherwise, the page may be re-assigned to other kernel components while it may still be mapped in the S-EPT.
KASAN detects use-after-free's like that. However, the TDX module code is not instrumented. It won't check against the KASAN state for it's accesses.
I had a brief chat about this with Dave and Kirill. A couple ideas were discussed. One was to use page_ext to keep a flag that says the page is in-use by the TDX module. There was also some discussion of using a normal page flag, and that the reserved page flag might prevent some of the MM operations that would be needed on guestmemfd pages. I didn't see the problem when I looked.
For the solution, basically the SEAMCALL wrappers set a flag when they hand a page to the TDX module, and clear it when they successfully reclaim it via tdh_mem_page_remove() or tdh_phymem_page_reclaim(). Then if the page makes it back to the page allocator, a warning is generated.
Also it was mentioned that SGX did have a similar issue to what is being worried about here:
https://lore.kernel.org/linux-sgx/[email protected]/T/#m86c8c4cf0e6b9a653bf0709a22bb360034a24d95
Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 17 lines
Quoted 4 lines
Quoted 12 lines
This would allow
guest_memfdto maintain an internal reference count for
each
private GFN. TDX would callguest_memfd_add_page_ref_count()for mapping
andguest_memfd_dec_page_ref_count()after a successful unmapping. Before truncating
a private page from the filemap,guest_memfdcould increase the real folio
reference count based on its internal reference count for the private GFN.Quoted 6 lines
What does this get us exactly? This is the argument to have less error prone
code that can survive forgetting to refcount on error? I don't see that it
is an
especially special case.Yes, for a less error prone code.
If this approach is considered too complex for an initial implementation, using
tdx_hold_page_on_error()is also a viable option.
Quoted 8 lines
Quoted 3 lines
I'm saying I don't think it's not a good enough reason. Why is it different then
other use-after free bugs? I feel like I'm missing something.By
tdx_hold_page_on_error(), it could be implememented as on removal failure, invoke aguest_memfdinterface to letguest_memfdknow exact ranges still being
under use by the TDX module due to unmapping failures.
Do you think it's ok?
Either way is ok to me. It seems like we have three ok solutions. But the tone of the thread is that we are solving some deep problem. Maybe I'm missing something.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 15 lines · Rick P
Quoted 7 lines · Yan Zhao
Quoted 7 lines
I don't think a potential bug in KVM is a good enough reason. If we are concerned can we think about a warning instead?
We had talked enhancing kasan to know when a page is mapped into S-EPT in the
past. So rather than design around potential bugs we could focus on having a simpler implementation with the infrastructure to catch and fix the bugs.Quoted 6 lines
However, if failing to remove a guest private page would only cause memory leak,
it's fine.
If TDX does not hold any refcount,guest_memfdhas to know that which private page is still mapped. Otherwise, the page may be re-assigned to other kernel components while it may still be mapped in the S-EPT.
Quoted 5 lines
KASAN detects use-after-free's like that. However, the TDX module code is not instrumented. It won't check against the KASAN state for it's accesses.
I had a brief chat about this with Dave and Kirill. A couple ideas were discussed. One was to use
page_extto keep a flag that says the page is in-use
Thanks!
To use page_ext, should we introduce a new flag PAGE_EXT_FIRMWARE_IN_USE, similar to PAGE_EXT_YOUNG?
Due to similar issues as those with normal page/folio flags (see the next comment for details), TDX needs to set PAGE_EXT_FIRMWARE_IN_USE on a page-by-page basis rather than folio-by-folio.
Additionally, it seems reasonable for guest_memfd not to copy the PAGE_EXT_FIRMWARE_IN_USE flag when splitting a huge folio? (in __folio_split() --> split_folio_to_order(), PAGE_EXT_YOUNG and PAGE_EXT_IDLE are copied to the new folios though).
Furthermore, page_ext uses extra memory. With CONFIG_64BIT, should we instead introduce a PG_firmware_in_use in page flags, similar to PG_young and PG_idle?
Quoted 8 lines
by the TDX module. There was also some discussion of using a normal page flag, and that the reserved page flag might prevent some of the MM operations that would be needed on guestmemfd pages. I didn't see the problem when I looked.
For the solution, basically the SEAMCALL wrappers set a flag when they hand a page to the TDX module, and clear it when they successfully reclaim it via
tdh_mem_page_remove()ortdh_phymem_page_reclaim(). Then if the page makes it back to the page allocator, a warning is generated.
After some testing, to use a normal page flag, we may need to set it on a page-by-page basis rather than folio-by-folio. See "Scheme 1". And guest_memfd may need to selectively copy page flags when splitting huge folios. See "Scheme 2".
Scheme 1: Set/unset page flag on folio-by-folio basis, i.e.
- set folio reserved at tdh_mem_page_aug(), tdh_mem_page_add(),
- unset folio reserved after a successful tdh_mem_page_remove() or tdh_phymem_page_reclaim().
It has problem in following scenario:
1. tdh_mem_page_aug() adds a 2MB folio. It marks the folio as reserved via "folio_set_reserved(page_folio(page))"
2. convert a 4KB page of the 2MB folio to shared. 2.1 tdh_mem_page_demote() is executed first.
2.2 tdh_mem_page_remove() then removes the 4KB mapping.
"folio_clear_reserved(page_folio(page))" clears reserved flag for the 2MB folio while the rest 511 pages are still mapped in the S-EPT.
2.3. guest_memfd splits the 2MB folio into 512 4KB folios.
Scheme 2: Set/unset page flag on page-by-page basis, i.e.
- set page flag reserved at tdh_mem_page_aug(), tdh_mem_page_add(),
- unset page flag reserved after a successful tdh_mem_page_remove() or
tdh_phymem_page_reclaim().
It has problem in following scenario:
1. tdh_mem_page_aug() adds a 2MB folio. It marks pages as reserved by
invoking "SetPageReserved()" on each page.
As the folio->flags equals to page[0]->flags, folio->flags is also
with reserved set.
2. convert a 4KB page of the 2MB folio to shared. say, it's page[4].
2.1 tdh_mem_page_demote() is executed first.
2.2 tdh_mem_page_remove() then removes the 4KB mapping.
"ClearPageReserved()" clears reserved flag of page[4] of the 2MB
folio.
2.3. guest_memfd splits the 2MB folio into 512 4KB folios.
In guestmem_hugetlb_split_folio(), "p->flags = folio->flags" marks
page[4]->flags as reserved again as page[0] is still reserved.
(see the code in https://lore.kernel.org/all/2ae41e0d80339da2b57011622ac2288fed65cd01.1747264138.git.ackerleytng@google.com/
for (i = 1; i < orig_nr_pages; ++i) {
struct page *p = folio_page(folio, i);
/* Copy flags from the first page to split pages. */
p->flags = folio->flags;
p->mapping = NULL;
clear_compound_head(p);
}
)I'm not sure if "p->flags = folio->flags" can be removed. Currently flag like PG_unevictable is preserved via this step.
If we selectively copy flags, we may need to implement the following changes to prevent the page from being available to the page allocator. Otherwise, the "HugePages_Free" count will not decrease, and the same huge folio will continue to be recycled (i.e., being allocated and consumed by other VMs).
mm/swap.c +25 -0
@@ -98,8 +98,36 @@ static void page_cache_release(struct folio *folio)unlock_page_lruvec_irqrestore (lruvec, flags); } +static inline bool folio_is_reserved (struct folio *folio) +{ +long nr_pages =folio_nr_pages (folio); +long i; + +for (i =0 ; i < nr_pages; i++) { +if (!PageReserved (folio_page (folio, i))) +continue ; + +return true; + } + +return false; +} +static void free_typed_folio (struct folio *folio) {
@@ -118,6 +146,13 @@ static void free_typed_folio(struct folio *folio)void __folio_put (struct folio *folio) { +if (folio_is_reserved (folio)) { +VM_WARN_ON_FOLIO (folio_is_reserved (folio), folio); +return ; + } +if (unlikely (folio_is_zone_device (folio))) {free_zone_device_folio (folio);return ;
@@ -986,6 +1021,12 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs)if (!folio_ref_sub_and_test (folio, nr_refs))continue ; +if (folio_is_reserved (folio)) { +VM_WARN_ON_FOLIO (folio_is_reserved (folio), folio); +continue ; + } +if (unlikely (folio_has_type (folio))) {/* typed folios have their own memcg, if any */ if (lruvec) {
Besides, guest_memfd needs to reject converting to shared when a page is still mapped in S-EPT.
virt/kvm/guest_memfd.c +40 -0
@@ -553,6 +553,41 @@ static void kvm_gmem_convert_invalidate_end(struct inode *inode,kvm_gmem_invalidate_end (gmem, invalidate_start, invalidate_end); } +static bool kvm_gmem_has_invalid_folio (struct address_space *mapping,pgoff_t start, +size_t nr_pages) +{ +pgoff_t index = start, end = start + nr_pages; +bool ret = false; + +while (index < end) { +struct folio *f; +long i =0 , nr; + + f =filemap_get_folio (mapping, index); +if (IS_ERR (f)) +continue ; + +if (f->index < start) + i = start - f->index; + + nr =folio_nr_pages (f); +if (f->index +folio_nr_pages (f) > end) + nr -= f->index +folio_nr_pages (f) - end; + +for (; i < nr; i++) { +if (PageReserved (folio_page (f, i))) { + ret = true; +folio_put (f); + goto out; + } + } + index +=folio_nr_pages (f); +folio_put (f); + } +out: +return ret; +}static int kvm_gmem_convert_should_proceed (struct inode *inode,struct conversion_work *work,bool to_shared,pgoff_t *error_index)
@@ -572,6 +607,12 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,if (ret)return ret;kvm_gmem_zap (gmem, work->start, work_end,KVM_FILTER_PRIVATE ); + +if (kvm_gmem_has_invalid_folio (inode->i_mapping, work->start, + work->nr_pages)) { + ret = -EFAULT ; + } + } }else { unmap_mapping_pages(inode->i_mapping, work->start,
Quoted 45 lines
Also it was mentioned that SGX did have a similar issue to what is being worried about here:
https://lore.kernel.org/linux-sgx/[email protected]/T/#m86c8c4cf0e6b9a653bf0709a22bb360034a24d95Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 17 lines
Quoted 4 lines
Quoted 12 lines
This would allow
guest_memfdto maintain an internal reference count for
each
private GFN. TDX would callguest_memfd_add_page_ref_count()for mapping
andguest_memfd_dec_page_ref_count()after a successful unmapping. Before truncating
a private page from the filemap,guest_memfdcould increase the real folio
reference count based on its internal reference count for the private GFN.Quoted 6 lines
What does this get us exactly? This is the argument to have less error prone
code that can survive forgetting to refcount on error? I don't see that it
is an
especially special case.Yes, for a less error prone code.
If this approach is considered too complex for an initial implementation, using
tdx_hold_page_on_error()is also a viable option.Quoted 8 lines
Quoted 3 lines
I'm saying I don't think it's not a good enough reason. Why is it different then
other use-after free bugs? I feel like I'm missing something.By
tdx_hold_page_on_error(), it could be implememented as on removal failure, invoke aguest_memfdinterface to letguest_memfdknow exact ranges still being
under use by the TDX module due to unmapping failures.
Do you think it's ok?Either way is ok to me. It seems like we have three ok solutions. But the tone of the thread is that we are solving some deep problem. Maybe I'm missing something.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 7 lines · Yan Zhao
To use
page_ext, should we introduce a new flagPAGE_EXT_FIRMWARE_IN_USE, similar toPAGE_EXT_YOUNG?Due to similar issues as those with normal page/folio flags (see the next comment for details), TDX needs to set
PAGE_EXT_FIRMWARE_IN_USEon a page-by-page basis rather than folio-by-folio.
Quoted 7 lines
Additionally, it seems reasonable for
guest_memfdnot to copy thePAGE_EXT_FIRMWARE_IN_USEflag when splitting a huge folio? (in__folio_split()-->split_folio_to_order(),PAGE_EXT_YOUNGandPAGE_EXT_IDLEare copied to the new folios though).Furthermore,
page_extuses extra memory. WithCONFIG_64BIT, should we instead introduce aPG_firmware_in_usein page flags, similar toPG_youngandPG_idle?
Page flags are a scarce resource. If we could have used an existing one, it would have been nice. But otherwise, I would guess the use case is not strong enough to justify adding one.
So PAGE_EXT_FIRMWARE_IN_USE is probably a better way to go. Due to the memory use, it would have to be a debug config like the others. If we have line of sight to a solution, how do you feel about the following direction to move past this issue:
1. Go with refcount on error approach for now (i.e. tdx_hold_page_on_error())
2. In a pfn-only future, plan to switch to guestmemfd callback instead of tdx_hold_page_on_error(). We don't understand the pfn-only feature enough to properly design for it anyway.
3. Plan for a PAGE_EXT_FIRMWARE_IN_USE as follow on work to huge pages. The reason why it should not be required before huge pages is because it is not necessary for correct code, only to catch incorrect code slipping in.
That is based on the assessment that the effort to change the zap path to communicate failure is too much churn. Do you happen to have a diffstat for a POC on this BTW?
Quoted 5 lines
Quoted 5 lines
by the TDX module. There was also some discussion of using a normal page flag, and that the reserved page flag might prevent some of the MM operations that would be needed on guestmemfd pages. I didn't see the problem when I looked.
Quoted 8 lines
Quoted 4 lines
For the solution, basically the SEAMCALL wrappers set a flag when they hand a page to the TDX module, and clear it when they successfully reclaim it via
tdh_mem_page_remove()ortdh_phymem_page_reclaim(). Then if the page makes it back to the page allocator, a warning is generated.After some testing, to use a normal page flag, we may need to set it on a page-by-page basis rather than folio-by-folio. See "Scheme 1". And
guest_memfdmay need to selectively copy page flags when splitting huge folios. See "Scheme 2".
With page_ext, it seems we could have it be per page from the beginning?
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 32 lines · Ackerley Tng
Quoted 30 lines · Yan Zhao
Quoted 15 lines · Rick P
Quoted 7 lines · Yan Zhao
Quoted 7 lines
I don't think a potential bug in KVM is a good enough reason. If we are concerned can we think about a warning instead?
We had talked enhancing kasan to know when a page is mapped into S-EPT in the
past. So rather than design around potential bugs we could focus on having a simpler implementation with the infrastructure to catch and fix the bugs.Quoted 6 lines
However, if failing to remove a guest private page would only cause memory leak,
it's fine.
If TDX does not hold any refcount,guest_memfdhas to know that which private page is still mapped. Otherwise, the page may be re-assigned to other kernel components while it may still be mapped in the S-EPT.Quoted 5 lines
KASAN detects use-after-free's like that. However, the TDX module code is not instrumented. It won't check against the KASAN state for it's accesses.
I had a brief chat about this with Dave and Kirill. A couple ideas were discussed. One was to use
page_extto keep a flag that says the page is in-useThanks!
To use
page_ext, should we introduce a new flagPAGE_EXT_FIRMWARE_IN_USE, similar toPAGE_EXT_YOUNG?Due to similar issues as those with normal page/folio flags (see the next comment for details), TDX needs to set
PAGE_EXT_FIRMWARE_IN_USEon a page-by-page basis rather than folio-by-folio.
Quoted 8 lines
Quoted 8 lines
Additionally, it seems reasonable for
guest_memfdnot to copy thePAGE_EXT_FIRMWARE_IN_USEflag when splitting a huge folio? (in__folio_split()-->split_folio_to_order(),PAGE_EXT_YOUNGandPAGE_EXT_IDLEare copied to the new folios though).Furthermore,
page_extuses extra memory. WithCONFIG_64BIT, should we instead introduce aPG_firmware_in_usein page flags, similar toPG_youngandPG_idle?
I think neither page flags nor page_ext will work for us, but see below.
Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
by the TDX module. There was also some discussion of using a normal page flag, and that the reserved page flag might prevent some of the MM operations that would be needed on guestmemfd pages. I didn't see the problem when I looked.
For the solution, basically the SEAMCALL wrappers set a flag when they hand a page to the TDX module, and clear it when they successfully reclaim it via
tdh_mem_page_remove()ortdh_phymem_page_reclaim(). Then if the page makes it back to the page allocator, a warning is generated.After some testing, to use a normal page flag, we may need to set it on a page-by-page basis rather than folio-by-folio. See "Scheme 1". And
guest_memfdmay need to selectively copy page flags when splitting huge folios. See "Scheme 2".Scheme 1: Set/unset page flag on folio-by-folio basis, i.e.
- set folio reserved attdh_mem_page_aug(),tdh_mem_page_add(),
- unset folio reserved after a successfultdh_mem_page_remove()ortdh_phymem_page_reclaim().It has problem in following scenario:
1.tdh_mem_page_aug()adds a 2MB folio. It marks the folio as reserved via "folio_set_reserved(page_folio(page))"2. convert a 4KB page of the 2MB folio to shared. 2.1
tdh_mem_page_demote()is executed first.
Quoted 8 lines
Quoted 8 lines
2.2
tdh_mem_page_remove()then removes the 4KB mapping.
"folio_clear_reserved(page_folio(page))" clears reserved flag for the 2MB folio while the rest 511 pages are still mapped in the S-EPT.2.3.
guest_memfdsplits the 2MB folio into 512 4KB folios.
Folio flags on their own won't work because they're not precise enough. A folio can be multiple 4K pages, and if a 4K page had failed to unmap, we want to be able to indicate which 4K page had the failure, instead of the entire folio. (But see below)
Quoted 26 lines
Quoted 26 lines
Scheme 2: Set/unset page flag on page-by-page basis, i.e.
- set page flag reserved attdh_mem_page_aug(),tdh_mem_page_add(),
- unset page flag reserved after a successfultdh_mem_page_remove()ortdh_phymem_page_reclaim().It has problem in following scenario:
1.tdh_mem_page_aug()adds a 2MB folio. It marks pages as reserved by invoking "SetPageReserved()" on each page.
As thefolio->flagsequals to page[0]->flags,folio->flagsis also with reserved set.2. convert a 4KB page of the 2MB folio to shared. say, it's page[4]. 2.1
tdh_mem_page_demote()is executed first.2.2
tdh_mem_page_remove()then removes the 4KB mapping.
"ClearPageReserved()" clears reserved flag of page[4] of the 2MB folio.2.3.
guest_memfdsplits the 2MB folio into 512 4KB folios.
Inguestmem_hugetlb_split_folio(), "p->flags=folio->flags" marks page[4]->flags as reserved again as page[0] is still reserved.(see the code in https://lore.kernel.org/all/2ae41e0d80339da2b57011622ac2288fed65cd01.1747264138.git.ackerleytng@google.com/
for (i = 1; i <orig_nr_pages; ++i) {
struct page *p =folio_page(folio, i);
Quoted 8 lines
Quoted 8 lines
/* Copy flags from the first page to split pages. */ p->flags = folio->flags; p->mapping = NULL; clear_compound_head(p); } )
Per-page flags won't work because we want to retain HugeTLB Vmemmap Optimization (HVO), which allows subsequent (identical) struct pages to alias to each other. If we use a per-page flag, then HVO would break since struct pages would no longer be identical to each other.
Quoted 1 line
Quoted 1 line
[...]
Let me try and summarize the current state of this discussion:
Topic 1: Does TDX need to somehow indicate that it is using a page?
This patch series uses refcounts to indicate that TDX is using a page, but that complicates private-to-shared conversions.
During a private-to-shared conversion, guest_memfd assumes that guest_memfd is trusted to manage private memory. TDX and other users should trust guest_memfd to keep the memory around.
Yan's position is that holding a refcount is in line with how IOMMU takes a refcount when a page is mapped into the IOMMU [1].
Yan had another suggestion, which is to indicate using a page flag [2].
I think we're in agreement that we don't want to have TDX hold a refcount while the page is mapped into the Secure EPTs, but taking a step back, do we really need to indicate (at all) that TDX is using a page?
In [3] Yan said
Quoted 5 lines
If TDX does not hold any refcount,
guest_memfdhas to know that which private
page is still mapped. Otherwise, the page may be re-assigned to other kernel
components while it may still be mapped in the S-EPT.
If the private page is mapped for regular VM use as private memory, guest_memfd is managing that, and the same page will not be re-assigned to any other kernel component. guest_memfd does hold refcounts in guest_memfd's filemap.
If the private page is still mapped because there was an unmapping failure, we can discuss that separately under error handling in Topic 2.
With this, can I confirm that we are in agreement that TDX does not need to indicate that it is using a page, and can trust guest_memfd to keep the page around for the VM?
Topic 2: How to handle unmapping/splitting errors arising from TDX?
Previously I was in favor of having unmap() return an error (Rick suggested doing a POC, and in a more recent email Rick asked for a diffstat), but Vishal and I talked about this and now I agree having unmapping return an error is not a good approach for these reasons.
1. Unmapping takes a range, and within the range there could be more than one unmapping error. I was previously thinking that unmap() could return 0 for success and the failed PFN on error. Returning a single PFN on error is okay-ish but if there are more errors it could get complicated.
Another error return option could be to return the folio where the unmapping/splitting issue happened, but that would not be sufficiently precise, since a folio could be larger than 4K and we want to track errors as precisely as we can to reduce memory loss due to errors.
2. What I think Yan has been trying to say: unmap() returning an error is non-standard in the kernel.
I think (1) is the dealbreaker here and there's no need to do the plumbing POC and diffstat.
So I think we're all in support of indicating unmapping/splitting issues without returning anything from unmap(), and the discussed options are
a. Refcounts: won't work - mostly discussed in this (sub-)thread
[3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.
b. Page flags: won't work with/can't benefit from HVO.
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually
had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.
d. Folio flags with folio splitting on error. This means that on
unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.
The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
e. Some other data structure in guest_memfd, say, a linked list, and a
function like kvm_gmem_add_error_pfn(struct page *page) that would look up the guest_memfd inode from the page and add the page's pfn to the linked list.
Everywhere in guest_memfd that does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.
Everywhere in guest_memfd that allocates pages will also check this linked list to make sure the pages are functional.
When guest_memfd truncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.
f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.
g. Like f, but basically treat an unmapping error as hardware poisoning.
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
[1] https://lore.kernel.org/all/aE%[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 119 lines · Ackerley Tng
Quoted 32 lines · Ackerley Tng
Quoted 30 lines · Yan Zhao
Quoted 15 lines · Rick P
Quoted 7 lines · Yan Zhao
Quoted 7 lines
I don't think a potential bug in KVM is a good enough reason. If we are concerned can we think about a warning instead?
We had talked enhancing kasan to know when a page is mapped into S-EPT in the
past. So rather than design around potential bugs we could focus on having a simpler implementation with the infrastructure to catch and fix the bugs.Quoted 6 lines
However, if failing to remove a guest private page would only cause memory leak,
it's fine.
If TDX does not hold any refcount,guest_memfdhas to know that which private page is still mapped. Otherwise, the page may be re-assigned to other kernel components while it may still be mapped in the S-EPT.Quoted 5 lines
KASAN detects use-after-free's like that. However, the TDX module code is not instrumented. It won't check against the KASAN state for it's accesses.
I had a brief chat about this with Dave and Kirill. A couple ideas were discussed. One was to use
page_extto keep a flag that says the page is in-useThanks!
To use
page_ext, should we introduce a new flagPAGE_EXT_FIRMWARE_IN_USE, similar toPAGE_EXT_YOUNG?Due to similar issues as those with normal page/folio flags (see the next comment for details), TDX needs to set
PAGE_EXT_FIRMWARE_IN_USEon a page-by-page basis rather than folio-by-folio.Quoted 8 lines
Quoted 8 lines
Additionally, it seems reasonable for
guest_memfdnot to copy thePAGE_EXT_FIRMWARE_IN_USEflag when splitting a huge folio? (in__folio_split()-->split_folio_to_order(),PAGE_EXT_YOUNGandPAGE_EXT_IDLEare copied to the new folios though).Furthermore,
page_extuses extra memory. WithCONFIG_64BIT, should we instead introduce aPG_firmware_in_usein page flags, similar toPG_youngandPG_idle?I think neither page flags nor
page_extwill work for us, but see below.Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
by the TDX module. There was also some discussion of using a normal page flag, and that the reserved page flag might prevent some of the MM operations that would be needed on guestmemfd pages. I didn't see the problem when I looked.
For the solution, basically the SEAMCALL wrappers set a flag when they hand a page to the TDX module, and clear it when they successfully reclaim it via
tdh_mem_page_remove()ortdh_phymem_page_reclaim(). Then if the page makes it back to the page allocator, a warning is generated.After some testing, to use a normal page flag, we may need to set it on a page-by-page basis rather than folio-by-folio. See "Scheme 1". And
guest_memfdmay need to selectively copy page flags when splitting huge folios. See "Scheme 2".Scheme 1: Set/unset page flag on folio-by-folio basis, i.e.
- set folio reserved attdh_mem_page_aug(),tdh_mem_page_add(),
- unset folio reserved after a successfultdh_mem_page_remove()ortdh_phymem_page_reclaim().It has problem in following scenario:
1.tdh_mem_page_aug()adds a 2MB folio. It marks the folio as reserved via "folio_set_reserved(page_folio(page))"2. convert a 4KB page of the 2MB folio to shared. 2.1
tdh_mem_page_demote()is executed first.Quoted 8 lines
Quoted 8 lines
2.2
tdh_mem_page_remove()then removes the 4KB mapping.
"folio_clear_reserved(page_folio(page))" clears reserved flag for the 2MB folio while the rest 511 pages are still mapped in the S-EPT.2.3.
guest_memfdsplits the 2MB folio into 512 4KB folios.Folio flags on their own won't work because they're not precise enough. A folio can be multiple 4K pages, and if a 4K page had failed to unmap, we want to be able to indicate which 4K page had the failure, instead of the entire folio. (But see below)
Quoted 26 lines
Quoted 26 lines
Scheme 2: Set/unset page flag on page-by-page basis, i.e.
- set page flag reserved attdh_mem_page_aug(),tdh_mem_page_add(),
- unset page flag reserved after a successfultdh_mem_page_remove()ortdh_phymem_page_reclaim().It has problem in following scenario:
1.tdh_mem_page_aug()adds a 2MB folio. It marks pages as reserved by invoking "SetPageReserved()" on each page.
As thefolio->flagsequals to page[0]->flags,folio->flagsis also with reserved set.2. convert a 4KB page of the 2MB folio to shared. say, it's page[4]. 2.1
tdh_mem_page_demote()is executed first.2.2
tdh_mem_page_remove()then removes the 4KB mapping.
"ClearPageReserved()" clears reserved flag of page[4] of the 2MB folio.2.3.
guest_memfdsplits the 2MB folio into 512 4KB folios.
Inguestmem_hugetlb_split_folio(), "p->flags=folio->flags" marks page[4]->flags as reserved again as page[0] is still reserved.(see the code in https://lore.kernel.org/all/2ae41e0d80339da2b57011622ac2288fed65cd01.1747264138.git.ackerleytng@google.com/
for (i = 1; i <orig_nr_pages; ++i) {
struct page *p =folio_page(folio, i);Quoted 8 lines
Quoted 8 lines
/* Copy flags from the first page to split pages. */ p->flags = folio->flags; p->mapping = NULL; clear_compound_head(p); } )
Quoted 4 lines
Per-page flags won't work because we want to retain HugeTLB Vmemmap Optimization (HVO), which allows subsequent (identical) struct pages to alias to each other. If we use a per-page flag, then HVO would break since struct pages would no longer be identical to each other.
Ah, I overlooked HVO.
In my testing, neither CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON nor hugetlb_free_vmemmap was set.
With HVO enabled, setting page flags on a per-page basis indeed does not work.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
[...]
Let me try and summarize the current state of this discussion:
Thanks for this summary.
Quoted 27 lines
Topic 1: Does TDX need to somehow indicate that it is using a page?
This patch series uses refcounts to indicate that TDX is using a page, but that complicates private-to-shared conversions.
During a private-to-shared conversion,
guest_memfdassumes thatguest_memfdis trusted to manage private memory. TDX and other users should trustguest_memfdto keep the memory around.Yan's position is that holding a refcount is in line with how IOMMU takes a refcount when a page is mapped into the IOMMU [1].
Yan had another suggestion, which is to indicate using a page flag [2].
I think we're in agreement that we don't want to have TDX hold a refcount while the page is mapped into the Secure EPTs, but taking a step back, do we really need to indicate (at all) that TDX is using a page?
In [3] Yan said
Quoted 5 lines
If TDX does not hold any refcount,
guest_memfdhas to know that which private
page is still mapped. Otherwise, the page may be re-assigned to other kernel
components while it may still be mapped in the S-EPT.
Quoted 4 lines
If the private page is mapped for regular VM use as private memory,
guest_memfdis managing that, and the same page will not be re-assigned to any other kernel component.guest_memfddoes hold refcounts in guest_memfd's filemap.
After kvm_gmem_release(), guest_memfd will return folios to hugetlb, so the same page could be re-assigned to other kernel components that allocate pages from hugetlb.
Quoted 7 lines
If the private page is still mapped because there was an unmapping failure, we can discuss that separately under error handling in Topic 2.
With this, can I confirm that we are in agreement that TDX does not need to indicate that it is using a page, and can trust
guest_memfdto keep the page around for the VM?
I thought it's not a must until I came across a comment from Sean:
"Should these bail early if the KVM_BUG_ON() is hit? Calling into the TDX module after bugging the VM is a bit odd."
https://lore.kernel.org/kvm/[email protected]/#t.
This comment refers to the following scenario:
when a 2MB non-leaf entry in the mirror root is zapped with shared mmu_lock, BUG_ON() will be triggered for TDX. But by the time handle_removed_pt() is reached, the 2MB non-leaf entry would have been successfully removed in the mirror root.
Bailing out early in remove_external_spte() would prevent the removal of 4KB private guest pages in the S-EPT later due to lacking of corresponding entry in the mirror root.
Since KVM MMU does not hold guest page's ref count, failing to notify TDX about the removal of a guest page could result in a situation where a page still mapped in the S-EPT is freed and re-allocated by the OS.
Therefore, indicating that TDX is using a page can be less error-prone, though it does consume more memory.
Quoted 32 lines
Topic 2: How to handle unmapping/splitting errors arising from TDX?
Previously I was in favor of having
unmap()return an error (Rick suggested doing a POC, and in a more recent email Rick asked for a diffstat), but Vishal and I talked about this and now I agree having unmapping return an error is not a good approach for these reasons.1. Unmapping takes a range, and within the range there could be more than one unmapping error. I was previously thinking that
unmap()could return 0 for success and the failed PFN on error. Returning a single PFN on error is okay-ish but if there are more errors it could get complicated.Another error return option could be to return the folio where the unmapping/splitting issue happened, but that would not be sufficiently precise, since a folio could be larger than 4K and we want to track errors as precisely as we can to reduce memory loss due to errors.
2. What I think Yan has been trying to say:
unmap()returning an error is non-standard in the kernel.I think (1) is the dealbreaker here and there's no need to do the plumbing POC and diffstat.
So I think we're all in support of indicating unmapping/splitting issues without returning anything from
unmap(), and the discussed options area. Refcounts: won't work - mostly discussed in this (sub-)thread
[3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.
Quoted 7 lines
b. Page flags: won't work with/can't benefit from HVO.
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually
had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.
For 1GB folios, more precise info will be better.
Quoted 7 lines
d. Folio flags with folio splitting on error. This means that on
unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
Could we restore the page structures before triggering unmap?
Quoted 13 lines
e. Some other data structure in
guest_memfd, say, a linked list, and a
function likekvm_gmem_add_error_pfn(struct page *page)that would look up theguest_memfdinode from the page and add the page's pfn to the linked list.Everywhere in
guest_memfdthat does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.Everywhere in
guest_memfdthat allocates pages will also check this linked list to make sure the pages are functional.
Quoted 6 lines
When
guest_memfdtruncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.
That seems like a good idea. If memory allocation for the linked list succeeds, mark the pages within a folio as troublesome; otherwise, mark the entire folio as troublesome.
But maybe c is good enough for 2MB folios.
Quoted 1 line
g. Like f, but basically treat an unmapping error as hardware poisoning.
Not sure if hwpoison bit can be used directly.
Further investigation is needed.
Quoted 9 lines
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
[1] https://lore.kernel.org/all/aE%[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 163 lines · Yan Zhao
Quoted 119 lines · Ackerley Tng
Quoted 32 lines · Ackerley Tng
Quoted 30 lines · Yan Zhao
Quoted 15 lines · Rick P
Quoted 7 lines · Yan Zhao
Quoted 7 lines
I don't think a potential bug in KVM is a good enough reason. If we are concerned can we think about a warning instead?
We had talked enhancing kasan to know when a page is mapped into S-EPT in the
past. So rather than design around potential bugs we could focus on having a simpler implementation with the infrastructure to catch and fix the bugs.Quoted 6 lines
However, if failing to remove a guest private page would only cause memory leak,
it's fine.
If TDX does not hold any refcount,guest_memfdhas to know that which private page is still mapped. Otherwise, the page may be re-assigned to other kernel components while it may still be mapped in the S-EPT.Quoted 5 lines
KASAN detects use-after-free's like that. However, the TDX module code is not instrumented. It won't check against the KASAN state for it's accesses.
I had a brief chat about this with Dave and Kirill. A couple ideas were discussed. One was to use
page_extto keep a flag that says the page is in-useThanks!
To use
page_ext, should we introduce a new flagPAGE_EXT_FIRMWARE_IN_USE, similar toPAGE_EXT_YOUNG?Due to similar issues as those with normal page/folio flags (see the next comment for details), TDX needs to set
PAGE_EXT_FIRMWARE_IN_USEon a page-by-page basis rather than folio-by-folio.Quoted 8 lines
Quoted 8 lines
Additionally, it seems reasonable for
guest_memfdnot to copy thePAGE_EXT_FIRMWARE_IN_USEflag when splitting a huge folio? (in__folio_split()-->split_folio_to_order(),PAGE_EXT_YOUNGandPAGE_EXT_IDLEare copied to the new folios though).Furthermore,
page_extuses extra memory. WithCONFIG_64BIT, should we instead introduce aPG_firmware_in_usein page flags, similar toPG_youngandPG_idle?I think neither page flags nor
page_extwill work for us, but see below.Quoted 25 lines
Quoted 25 lines
Quoted 8 lines
by the TDX module. There was also some discussion of using a normal page flag, and that the reserved page flag might prevent some of the MM operations that would be needed on guestmemfd pages. I didn't see the problem when I looked.
For the solution, basically the SEAMCALL wrappers set a flag when they hand a page to the TDX module, and clear it when they successfully reclaim it via
tdh_mem_page_remove()ortdh_phymem_page_reclaim(). Then if the page makes it back to the page allocator, a warning is generated.After some testing, to use a normal page flag, we may need to set it on a page-by-page basis rather than folio-by-folio. See "Scheme 1". And
guest_memfdmay need to selectively copy page flags when splitting huge folios. See "Scheme 2".Scheme 1: Set/unset page flag on folio-by-folio basis, i.e.
- set folio reserved attdh_mem_page_aug(),tdh_mem_page_add(),
- unset folio reserved after a successfultdh_mem_page_remove()ortdh_phymem_page_reclaim().It has problem in following scenario:
1.tdh_mem_page_aug()adds a 2MB folio. It marks the folio as reserved via "folio_set_reserved(page_folio(page))"2. convert a 4KB page of the 2MB folio to shared. 2.1
tdh_mem_page_demote()is executed first.Quoted 8 lines
Quoted 8 lines
2.2
tdh_mem_page_remove()then removes the 4KB mapping.
"folio_clear_reserved(page_folio(page))" clears reserved flag for the 2MB folio while the rest 511 pages are still mapped in the S-EPT.2.3.
guest_memfdsplits the 2MB folio into 512 4KB folios.Folio flags on their own won't work because they're not precise enough. A folio can be multiple 4K pages, and if a 4K page had failed to unmap, we want to be able to indicate which 4K page had the failure, instead of the entire folio. (But see below)
Quoted 26 lines
Quoted 26 lines
Scheme 2: Set/unset page flag on page-by-page basis, i.e.
- set page flag reserved attdh_mem_page_aug(),tdh_mem_page_add(),
- unset page flag reserved after a successfultdh_mem_page_remove()ortdh_phymem_page_reclaim().It has problem in following scenario:
1.tdh_mem_page_aug()adds a 2MB folio. It marks pages as reserved by invoking "SetPageReserved()" on each page.
As thefolio->flagsequals to page[0]->flags,folio->flagsis also with reserved set.2. convert a 4KB page of the 2MB folio to shared. say, it's page[4]. 2.1
tdh_mem_page_demote()is executed first.2.2
tdh_mem_page_remove()then removes the 4KB mapping.
"ClearPageReserved()" clears reserved flag of page[4] of the 2MB folio.2.3.
guest_memfdsplits the 2MB folio into 512 4KB folios.
Inguestmem_hugetlb_split_folio(), "p->flags=folio->flags" marks page[4]->flags as reserved again as page[0] is still reserved.(see the code in https://lore.kernel.org/all/2ae41e0d80339da2b57011622ac2288fed65cd01.1747264138.git.ackerleytng@google.com/
for (i = 1; i <orig_nr_pages; ++i) {
struct page *p =folio_page(folio, i);Quoted 8 lines
Quoted 8 lines
/* Copy flags from the first page to split pages. */ p->flags = folio->flags; p->mapping = NULL; clear_compound_head(p); } )Quoted 4 lines
Per-page flags won't work because we want to retain HugeTLB Vmemmap Optimization (HVO), which allows subsequent (identical) struct pages to alias to each other. If we use a per-page flag, then HVO would break since struct pages would no longer be identical to each other.
Ah, I overlooked HVO.
In my testing, neitherCONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ONnorhugetlb_free_vmemmapwas set.With HVO enabled, setting page flags on a per-page basis indeed does not work.
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
[...]
Let me try and summarize the current state of this discussion:
Thanks for this summary.
Quoted 27 lines
Topic 1: Does TDX need to somehow indicate that it is using a page?
This patch series uses refcounts to indicate that TDX is using a page, but that complicates private-to-shared conversions.
During a private-to-shared conversion,
guest_memfdassumes thatguest_memfdis trusted to manage private memory. TDX and other users should trustguest_memfdto keep the memory around.Yan's position is that holding a refcount is in line with how IOMMU takes a refcount when a page is mapped into the IOMMU [1].
Yan had another suggestion, which is to indicate using a page flag [2].
I think we're in agreement that we don't want to have TDX hold a refcount while the page is mapped into the Secure EPTs, but taking a step back, do we really need to indicate (at all) that TDX is using a page?
In [3] Yan said
Quoted 5 lines
If TDX does not hold any refcount,
guest_memfdhas to know that which private
page is still mapped. Otherwise, the page may be re-assigned to other kernel
components while it may still be mapped in the S-EPT.
Quoted 8 lines
Quoted 4 lines
If the private page is mapped for regular VM use as private memory,
guest_memfdis managing that, and the same page will not be re-assigned to any other kernel component.guest_memfddoes hold refcounts in guest_memfd's filemap.After
kvm_gmem_release(),guest_memfdwill return folios to hugetlb, so the same page could be re-assigned to other kernel components that allocate pages from hugetlb.
Yes, at kvm_gmem_release(), guest_memfd also unmaps all the private and shared pages from Secure EPTs, so on successful unmap, all is good, the page can and should be reused elsewhere.
On unsuccessful unmap, we can then handle that according to the option we pick from below. e.g. if we pick options f or g, the page should never be reused elsewhere.
If we pick other options, then we would have to ensure the page doesn't get used elsewhere anyway, so TDX still won't need to hold a refcount while using the page?
Quoted 22 lines
Quoted 7 lines
If the private page is still mapped because there was an unmapping failure, we can discuss that separately under error handling in Topic 2.
With this, can I confirm that we are in agreement that TDX does not need to indicate that it is using a page, and can trust
guest_memfdto keep the page around for the VM?I thought it's not a must until I came across a comment from Sean:
"Should these bail early if theKVM_BUG_ON()is hit? Calling into the TDX module after bugging the VM is a bit odd."
https://lore.kernel.org/kvm/[email protected]/#t.This comment refers to the following scenario:
when a 2MB non-leaf entry in the mirror root is zapped with sharedmmu_lock,BUG_ON()will be triggered for TDX. But by the timehandle_removed_pt()is reached, the 2MB non-leaf entry would have been successfully removed in the mirror root.Bailing out early in
remove_external_spte()would prevent the removal of 4KB private guest pages in the S-EPT later due to lacking of corresponding entry in the mirror root.
Quoted 7 lines
Since KVM MMU does not hold guest page's ref count, failing to notify TDX about the removal of a guest page could result in a situation where a page still mapped in the S-EPT is freed and re-allocated by the OS.
Therefore, indicating that TDX is using a page can be less error-prone, though it does consume more memory.
This is true, but instead of somehow indicating that the page is used by TDX upfront, how about treating this as an error, so we should not bail out before first indicating an error on the page, such as using Option f or g below. Indicating an error on the page will prevent the case where a page still mapped in the S-EPT is freed and reused by the OS elsewhere.
Quoted 42 lines
Quoted 32 lines
Topic 2: How to handle unmapping/splitting errors arising from TDX?
Previously I was in favor of having
unmap()return an error (Rick suggested doing a POC, and in a more recent email Rick asked for a diffstat), but Vishal and I talked about this and now I agree having unmapping return an error is not a good approach for these reasons.1. Unmapping takes a range, and within the range there could be more than one unmapping error. I was previously thinking that
unmap()could return 0 for success and the failed PFN on error. Returning a single PFN on error is okay-ish but if there are more errors it could get complicated.Another error return option could be to return the folio where the unmapping/splitting issue happened, but that would not be sufficiently precise, since a folio could be larger than 4K and we want to track errors as precisely as we can to reduce memory loss due to errors.
2. What I think Yan has been trying to say:
unmap()returning an error is non-standard in the kernel.I think (1) is the dealbreaker here and there's no need to do the plumbing POC and diffstat.
So I think we're all in support of indicating unmapping/splitting issues without returning anything from
unmap(), and the discussed options area. Refcounts: won't work - mostly discussed in this (sub-)thread
[3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.Quoted 7 lines
b. Page flags: won't work with/can't benefit from HVO.
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually
had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.For 1GB folios, more precise info will be better.
Quoted 9 lines
Quoted 7 lines
d. Folio flags with folio splitting on error. This means that on
unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
Could we restore the page structures before triggering unmap?
Do you mean every time, before unmapping (every conversion, truncation, etc), first restore the page structs in preparation for unmap failure, then re-optimize HVO after successful unmap?
Restoring the page structures is quite an expensive operation IIUC, involving time and allocations, and it kind of defeats the purpose of HVO if we need to keep extra memory around to keep restoring and undoing HVO.
Or do you mean only on unmap failure, undo restore page structs to mark the error?
Because it is expensive, it seems hard for this to work when the unmap fails (the machine might be in some dire state already).
Quoted 16 lines
Quoted 16 lines
e. Some other data structure in
guest_memfd, say, a linked list, and a
function likekvm_gmem_add_error_pfn(struct page *page)that would look up theguest_memfdinode from the page and add the page's pfn to the linked list.Everywhere in
guest_memfdthat does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.Everywhere in
guest_memfdthat allocates pages will also check this linked list to make sure the pages are functional.When
guest_memfdtruncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.
Quoted 7 lines
Quoted 3 lines
f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.That seems like a good idea. If memory allocation for the linked list succeeds, mark the pages within a folio as troublesome; otherwise, mark the entire folio as troublesome.
This is already what HugeTLB does for poisoning in folio_set_hugetlb_hwpoison(), so if an unmapping error can be treated as hardware poisoning, we could re-use all that infrastructure.
Quoted 2 lines
But maybe c is good enough for 2MB folios.
I think it'd be awkward/troublesome to treat different sizes of folios differently and I would prefer not to do this.
Quoted 4 lines
Quoted 1 line
g. Like f, but basically treat an unmapping error as hardware poisoning.
Not sure if hwpoison bit can be used directly.
Further investigation is needed.
Would appreciate your assessment on whether an unmapping error can be treated as hardware poisoning!
Quoted 5 lines
Quoted 5 lines
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
I have another option h to add: if there is a unmapping error from TDX, can it be an indication of compromise, in terms of security? Should TDX continue to be trusted to run the TD or other TDs securely? If there is some unmapping error, could correctness in the entire host be in question?
If either correctness or security is broken, would it be acceptable to do a full BUG_ON and crash the system, since neither TDX nor regular VMs on the host should trusted to run correctly after this kind of error?
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 5 lines · Ackerley Tng
I have another option h to add: if there is a unmapping error from TDX, can it be an indication of compromise, in terms of security? Should TDX continue to be trusted to run the TD or other TDs securely? If there is some unmapping error, could correctness in the entire host be in question?
Maybe, but it's the TDX module's job to do something about this. The threat model of TDX doesn't involve the host VMM ensuring integrity of the TD.
Quoted 4 lines
If either correctness or security is broken, would it be acceptable to do a full
BUG_ONand crash the system, since neither TDX nor regular VMs on the host should trusted to run correctly after this kind of error?
BUG_ON() won't be acceptable. See Linus' opinion on the subject. The standard practice is to warn and let people run panic_on_warn if they want to be paranoid. And we already will generate a warning so it's possible to configure for this behavior today.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Ackerley TngQuoted 34 lines · Ackerley Tng
Let me try and summarize the current state of this discussion:
Topic 1: Does TDX need to somehow indicate that it is using a page?
This patch series uses refcounts to indicate that TDX is using a page, but that complicates private-to-shared conversions.
During a private-to-shared conversion,
guest_memfdassumes thatguest_memfdis trusted to manage private memory. TDX and other users should trustguest_memfdto keep the memory around.Yan's position is that holding a refcount is in line with how IOMMU takes a refcount when a page is mapped into the IOMMU [1].
Yan had another suggestion, which is to indicate using a page flag [2].
I think we're in agreement that we don't want to have TDX hold a refcount while the page is mapped into the Secure EPTs, but taking a step back, do we really need to indicate (at all) that TDX is using a page?
In [3] Yan said
Quoted 5 lines
If TDX does not hold any refcount,
guest_memfdhas to know that which private
page is still mapped. Otherwise, the page may be re-assigned to other kernel
components while it may still be mapped in the S-EPT.If the private page is mapped for regular VM use as private memory,
guest_memfdis managing that, and the same page will not be re-assigned to any other kernel component.guest_memfddoes hold refcounts in guest_memfd's filemap.
Quoted 6 lines
If the private page is still mapped because there was an unmapping failure, we can discuss that separately under error handling in Topic 2.
With this, can I confirm that we are in agreement that TDX does not need to indicate that it is using a page, and can trust
guest_memfdto keep the page around for the VM?
Minor correction here. Yan was concerned about *bugs* happening when freeing pages that are accidentally still mapped in the S-EPT. My opinion is that this is not especially risky to happen here vs other similar places, but it could be helpful if there was a way to catch such bugs. The page flag, or page_ext direction came out of a discussion with Dave and Kirill. If it could run all the time that would be great, but if not a debug config could be sufficient. For example like CONFIG_PAGE_TABLE_CHECK. It doesn't need to support vmemmap optimizations because the debug checking doesn't need to run all the time. Overhead for debug settings is very normal.
Quoted 7 lines
Topic 2: How to handle unmapping/splitting errors arising from TDX?
Previously I was in favor of having
unmap()return an error (Rick suggested doing a POC, and in a more recent email Rick asked for a diffstat), but Vishal and I talked about this and now I agree having unmapping return an error is not a good approach for these reasons.
Ok, let's close this option then.
Quoted 19 lines
1. Unmapping takes a range, and within the range there could be more than one unmapping error. I was previously thinking that
unmap()could return 0 for success and the failed PFN on error. Returning a single PFN on error is okay-ish but if there are more errors it could get complicated.Another error return option could be to return the folio where the unmapping/splitting issue happened, but that would not be sufficiently precise, since a folio could be larger than 4K and we want to track errors as precisely as we can to reduce memory loss due to errors.
2. What I think Yan has been trying to say:
unmap()returning an error is non-standard in the kernel.I think (1) is the dealbreaker here and there's no need to do the plumbing POC and diffstat.
Quoted 8 lines
So I think we're all in support of indicating unmapping/splitting issues without returning anything from
unmap(), and the discussed options area. Refcounts: won't work - mostly discussed in this (sub-)thread [3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.
b. Page flags: won't work with/can't benefit from HVO.
As above, this was for the purpose of catching bugs, not for guestmemfd to logically depend on it.
Quoted 6 lines
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.
For a scenario of TDX module bug, it seems ok to me.
Quoted 18 lines
d. Folio flags with folio splitting on error. This means that on unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.
The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
e. Some other data structure in
guest_memfd, say, a linked list, and a function likekvm_gmem_add_error_pfn(struct page *page)that would look up theguest_memfdinode from the page and add the page's pfn to the linked list.Everywhere in
guest_memfdthat does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.
Quoted 5 lines
Everywhere in
guest_memfdthat allocates pages will also check this linked list to make sure the pages are functional.When
guest_memfdtruncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.
I think this is a fine option.
Quoted 10 lines
f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.g. Like f, but basically treat an unmapping error as hardware poisoning.
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
What does HWPOISON bring over refcounting the page/folio so that it never returns to the page allocator? We are bugging the TD in these cases. Ohhh... Is this about the code to allow gmem fds to be handed to new VMs?
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 9 lines · Rick Edgecombe
Minor correction here. Yan was concerned about *bugs* happening when freeing pages that are accidentally still mapped in the S-EPT. My opinion is that this is not especially risky to happen here vs other similar places, but it could be helpful if there was a way to catch such bugs. The page flag, or
page_extdirection came out of a discussion with Dave and Kirill. If it could run all the time that would be great, but if not a debug config could be sufficient. For example likeCONFIG_PAGE_TABLE_CHECK. It doesn't need to support vmemmap optimizations because the debug checking doesn't need to run all the time. Overhead for debug settings is very normal.
Note, this is separate from the problem of how to handle or notify TDX unmap errors. That is still an open regardless. But Yan was concerned if we didn't take a reference when we first mapped it, that it could be more error prone. So this flag was an alternative to *holding* a reference during the lifetime of S- EPT mapping.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Edgecombe, Rick PQuoted 42 lines · Rick P
Quoted 34 lines · Ackerley Tng
Let me try and summarize the current state of this discussion:
Topic 1: Does TDX need to somehow indicate that it is using a page?
This patch series uses refcounts to indicate that TDX is using a page, but that complicates private-to-shared conversions.
During a private-to-shared conversion,
guest_memfdassumes thatguest_memfdis trusted to manage private memory. TDX and other users should trustguest_memfdto keep the memory around.Yan's position is that holding a refcount is in line with how IOMMU takes a refcount when a page is mapped into the IOMMU [1].
Yan had another suggestion, which is to indicate using a page flag [2].
I think we're in agreement that we don't want to have TDX hold a refcount while the page is mapped into the Secure EPTs, but taking a step back, do we really need to indicate (at all) that TDX is using a page?
In [3] Yan said
Quoted 5 lines
If TDX does not hold any refcount,
guest_memfdhas to know that which private
page is still mapped. Otherwise, the page may be re-assigned to other kernel
components while it may still be mapped in the S-EPT.If the private page is mapped for regular VM use as private memory,
guest_memfdis managing that, and the same page will not be re-assigned to any other kernel component.guest_memfddoes hold refcounts in guest_memfd's filemap.Quoted 6 lines
If the private page is still mapped because there was an unmapping failure, we can discuss that separately under error handling in Topic 2.
With this, can I confirm that we are in agreement that TDX does not need to indicate that it is using a page, and can trust
guest_memfdto keep the page around for the VM?
Quoted 10 lines
Minor correction here. Yan was concerned about *bugs* happening when freeing pages that are accidentally still mapped in the S-EPT. My opinion is that this is not especially risky to happen here vs other similar places, but it could be helpful if there was a way to catch such bugs. The page flag, or
page_extdirection came out of a discussion with Dave and Kirill. If it could run all the time that would be great, but if not a debug config could be sufficient. For example likeCONFIG_PAGE_TABLE_CHECK. It doesn't need to support vmemmap optimizations because the debug checking doesn't need to run all the time. Overhead for debug settings is very normal.
I see, let's call debug checking Topic 3 then, to separate it from Topic 1, which is TDX indicating that it is using a page for production kernels.
Topic 3: How should TDX indicate use of a page for debugging?
I'm okay if for debugging, TDX uses anything other than refcounts for checking, because refcounts will interfere with conversions.
Rick's other email is correct. The correct link should be
https://lore.kernel.org/all/[email protected]/.
[INTERFERE WITH CONVERSIONS]
To summarize, if TDX uses refcounts to indicate that it is using a page, or to indicate anything else, then we cannot easily split a page on private to shared conversions.
Specifically, consider the case where only the x-th subpage of a huge folio is mapped into Secure-EPTs. When the guest requests to convert some subpage to shared, the huge folio has to be split for core-mm. Core-mm, which will use the shared page, must have split folios to be able to accurately and separately track refcounts for subpages.
During splitting, guest_memfd would see refcount of 512 (for 2M page being in the filemap) + 1 (if TDX indicates that the x-th subpage is mapped using a refcount), but would not be able to tell that the 513th refcount belongs to the x-th subpage. guest_memfd can't split the huge folio unless it knows how to distribute the 513th refcount.
One might say guest_memfd could clear all the refcounts that TDX is holding on the huge folio by unmapping the entire huge folio from the Secure-EPTs, but unmapping the entire huge folio for TDX means zeroing the contents and requiring guest re-acceptance. Both of these would mess up guest operation.
Hence, guest_memfd's solution is to require that users of guest_memfd for private memory trust guest_memfd to maintain the pages around and not take any refcounts.
So back to Topic 1, for production kernels, is it okay that TDX does not need to indicate that it is using a page, and can trust guest_memfd to keep the page around for the VM?
Quoted 81 lines
Quoted 7 lines
Topic 2: How to handle unmapping/splitting errors arising from TDX?
Previously I was in favor of having
unmap()return an error (Rick suggested doing a POC, and in a more recent email Rick asked for a diffstat), but Vishal and I talked about this and now I agree having unmapping return an error is not a good approach for these reasons.Ok, let's close this option then.
Quoted 19 lines
1. Unmapping takes a range, and within the range there could be more than one unmapping error. I was previously thinking that
unmap()could return 0 for success and the failed PFN on error. Returning a single PFN on error is okay-ish but if there are more errors it could get complicated.Another error return option could be to return the folio where the unmapping/splitting issue happened, but that would not be sufficiently precise, since a folio could be larger than 4K and we want to track errors as precisely as we can to reduce memory loss due to errors.
2. What I think Yan has been trying to say:
unmap()returning an error is non-standard in the kernel.I think (1) is the dealbreaker here and there's no need to do the plumbing POC and diffstat.
Quoted 8 lines
So I think we're all in support of indicating unmapping/splitting issues without returning anything from
unmap(), and the discussed options area. Refcounts: won't work - mostly discussed in this (sub-)thread [3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.
b. Page flags: won't work with/can't benefit from HVO.
As above, this was for the purpose of catching bugs, not for guestmemfd to logically depend on it.
Quoted 6 lines
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.
For a scenario of TDX module bug, it seems ok to me.
Quoted 18 lines
d. Folio flags with folio splitting on error. This means that on unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.
The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
e. Some other data structure in
guest_memfd, say, a linked list, and a function likekvm_gmem_add_error_pfn(struct page *page)that would look up theguest_memfdinode from the page and add the page's pfn to the linked list.Everywhere in
guest_memfdthat does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.Quoted 5 lines
Everywhere in
guest_memfdthat allocates pages will also check this linked list to make sure the pages are functional.When
guest_memfdtruncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.I think this is a fine option.
Quoted 5 lines
f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.
Quoted 8 lines
Quoted 5 lines
g. Like f, but basically treat an unmapping error as hardware poisoning.
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
What does HWPOISON bring over refcounting the page/folio so that it never returns to the page allocator?
For Topic 2 (handling TDX unmapping errors), HWPOISON is better than refcounting because refcounting interferes with conversions (see [INTERFERE WITH CONVERSIONS] above).
Quoted 1 line
We are bugging the TD in these cases.
Bugging the TD does not help to prevent future conversions from being interfered with.
1. Conversions involves unmapping, so we could actually be in a conversion, the unmapping is performed and fails, and then we try to split and enter an infinite loop since private to shared conversions assumes guest_memfd holds the only refcounts on guest_memfd memory.
2. The conversion ioctl is a guest_memfd ioctl, not a VM ioctl, and so there is no check that the VM is not dead. There shouldn't be any check on the VM, because shareability is a property of the memory and should be changeable independent of the associated VM.
Quoted 2 lines
Ohhh... Is
this about the code to allow gmem fds to be handed to new VMs?
Nope, it's not related to linking. The proposed KVM_LINK_GUEST_MEMFD ioctl [4] also doesn't check if the source VM is dead. There shouldn't be any check on the source VM, since the memory is from guest_memfd and should be independently transferable to a new VM.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 8 lines · Ackerley Tng
I see, let's call debug checking Topic 3 then, to separate it from Topic 1, which is TDX indicating that it is using a page for production kernels.
Topic 3: How should TDX indicate use of a page for debugging?
I'm okay if for debugging, TDX uses anything other than refcounts for checking, because refcounts will interfere with conversions.
Ok. It can be follow on work I think.
Quoted 28 lines
Rick's other email is correct. The correct link should be
https://lore.kernel.org/all/[email protected]/.[INTERFERE WITH CONVERSIONS]
To summarize, if TDX uses refcounts to indicate that it is using a page, or to indicate anything else, then we cannot easily split a page on private to shared conversions.
Specifically, consider the case where only the x-th subpage of a huge folio is mapped into Secure-EPTs. When the guest requests to convert some subpage to shared, the huge folio has to be split for core-mm. Core-mm, which will use the shared page, must have split folios to be able to accurately and separately track refcounts for subpages.
During splitting,
guest_memfdwould see refcount of 512 (for 2M page being in the filemap) + 1 (if TDX indicates that the x-th subpage is mapped using a refcount), but would not be able to tell that the 513th refcount belongs to the x-th subpage.guest_memfdcan't split the huge folio unless it knows how to distribute the 513th refcount.One might say
guest_memfdcould clear all the refcounts that TDX is holding on the huge folio by unmapping the entire huge folio from the Secure-EPTs, but unmapping the entire huge folio for TDX means zeroing the contents and requiring guest re-acceptance. Both of these would mess up guest operation.
Quoted 7 lines
Hence, guest_memfd's solution is to require that users of
guest_memfdfor private memory trustguest_memfdto maintain the pages around and not take any refcounts.So back to Topic 1, for production kernels, is it okay that TDX does not need to indicate that it is using a page, and can trust
guest_memfdto keep the page around for the VM?
I think Yan's concern is not totally invalid. But I don't see a problem if we have a line of sight to adding debug checking as follow on work. That is kind of the path I was trying to nudge.
Quoted 88 lines
Quoted 88 lines
Quoted 8 lines
Topic 2: How to handle unmapping/splitting errors arising from TDX?
Previously I was in favor of having
unmap()return an error (Rick suggested doing a POC, and in a more recent email Rick asked for a diffstat), but Vishal and I talked about this and now I agree having unmapping return an error is not a good approach for these reasons.Ok, let's close this option then.
Quoted 19 lines
1. Unmapping takes a range, and within the range there could be more than one unmapping error. I was previously thinking that
unmap()could return 0 for success and the failed PFN on error. Returning a single PFN on error is okay-ish but if there are more errors it could get complicated.Another error return option could be to return the folio where the unmapping/splitting issue happened, but that would not be sufficiently precise, since a folio could be larger than 4K and we want to track errors as precisely as we can to reduce memory loss due to errors.
2. What I think Yan has been trying to say:
unmap()returning an error is non-standard in the kernel.I think (1) is the dealbreaker here and there's no need to do the plumbing POC and diffstat.
Quoted 8 lines
So I think we're all in support of indicating unmapping/splitting issues without returning anything from
unmap(), and the discussed options area. Refcounts: won't work - mostly discussed in this (sub-)thread [3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.
b. Page flags: won't work with/can't benefit from HVO.
As above, this was for the purpose of catching bugs, not for guestmemfd to logically depend on it.
Quoted 6 lines
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.
For a scenario of TDX module bug, it seems ok to me.
Quoted 18 lines
d. Folio flags with folio splitting on error. This means that on unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.
The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
e. Some other data structure in
guest_memfd, say, a linked list, and a function likekvm_gmem_add_error_pfn(struct page *page)that would look up theguest_memfdinode from the page and add the page's pfn to the linked list.Everywhere in
guest_memfdthat does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.Quoted 5 lines
Everywhere in
guest_memfdthat allocates pages will also check this linked list to make sure the pages are functional.When
guest_memfdtruncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.I think this is a fine option.
Quoted 10 lines
f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.g. Like f, but basically treat an unmapping error as hardware poisoning.
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
Quoted 6 lines
Quoted 2 lines
What does HWPOISON bring over refcounting the page/folio so that it never returns to the page allocator?
For Topic 2 (handling TDX unmapping errors), HWPOISON is better than refcounting because refcounting interferes with conversions (see [INTERFERE WITH CONVERSIONS] above).
I don't know if it quite fits. I think it would be better to not pollute the concept if possible.
Quoted 11 lines
Quoted 2 lines
We are bugging the TD in these cases.
Bugging the TD does not help to prevent future conversions from being interfered with.
1. Conversions involves unmapping, so we could actually be in a conversion, the unmapping is performed and fails, and then we try to split and enter an infinite loop since private to shared conversions assumes
guest_memfdholds the only refcounts onguest_memfdmemory.
Quoted 4 lines
2. The conversion ioctl is a
guest_memfdioctl, not a VM ioctl, and so there is no check that the VM is not dead. There shouldn't be any check on the VM, because shareability is a property of the memory and should be changeable independent of the associated VM.
Hmm, they are both about unlinking guestmemfd from a VM lifecycle then. Is that a better way to put it?
Quoted 8 lines
Quoted 3 lines
Ohhh... Is
this about the code to allow gmem fds to be handed to new VMs?Nope, it's not related to linking. The proposed
KVM_LINK_GUEST_MEMFDioctl [4] also doesn't check if the source VM is dead. There shouldn't be any check on the source VM, since the memory is fromguest_memfdand should be independently transferable to a new VM.
If a page is mapped in the old TD, and a new TD is started, re-mapping the same page should be prevented somehow, right?
It really does seem like guestmemfd is the right place to keep the the "stuck page" state. If guestmemfd is not tied to a VM and can be re-used, it should be the one to decide whether they can be mapped again. Refcounting on error is about preventing return to the page allocator but that is not the only problem.
I do think that these threads have gone on far too long. It's probably about time to move forward with something even if it's just to have something to discuss that doesn't require footnoting so many lore links. So how about we move forward with option e as a next step. Does that sound good Yan?
Ackerley, thank you very much for pulling together this summary.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 60 lines · Rick P
Quoted 60 lines
Quoted 60 lines
Quoted 8 lines
So I think we're all in support of indicating unmapping/splitting issues without returning anything from
unmap(), and the discussed options area. Refcounts: won't work - mostly discussed in this (sub-)thread [3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.
b. Page flags: won't work with/can't benefit from HVO.
As above, this was for the purpose of catching bugs, not for guestmemfd to logically depend on it.
Quoted 6 lines
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.
For a scenario of TDX module bug, it seems ok to me.
Quoted 18 lines
d. Folio flags with folio splitting on error. This means that on unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.
The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
e. Some other data structure in
guest_memfd, say, a linked list, and a function likekvm_gmem_add_error_pfn(struct page *page)that would look up theguest_memfdinode from the page and add the page's pfn to the linked list.Everywhere in
guest_memfdthat does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.Quoted 5 lines
Everywhere in
guest_memfdthat allocates pages will also check this linked list to make sure the pages are functional.When
guest_memfdtruncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.I think this is a fine option.
Quoted 10 lines
f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.g. Like f, but basically treat an unmapping error as hardware poisoning.
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
What does HWPOISON bring over refcounting the page/folio so that it never returns to the page allocator?
...
Quoted 4 lines
I do think that these threads have gone on far too long. It's probably about time to move forward with something even if it's just to have something to discuss that doesn't require footnoting so many lore links. So how about we move forward with option e as a next step. Does that sound good Yan?
I'm ok with e if allocation of memory for the linked list is not a problem. Otherwise, I feel that a simpler solution would be to set a folio flag when an unmapping error occurs.
guest_memfd needs to check this folio flag before the actual conversion and in kvm_gmem_free_folio().
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Edgecombe, Rick PQuoted 5 lines · Rick P
Quoted 4 lines · Ackerley Tng
I see, let's call debug checking Topic 3 then, to separate it from Topic 1, which is TDX indicating that it is using a page for production kernels.
Quoted 7 lines
Quoted 4 lines
Topic 3: How should TDX indicate use of a page for debugging?
I'm okay if for debugging, TDX uses anything other than refcounts for checking, because refcounts will interfere with conversions.
Ok. It can be follow on work I think.
Yup I agree.
Quoted 131 lines
Quoted 28 lines
Rick's other email is correct. The correct link should be
https://lore.kernel.org/all/[email protected]/.[INTERFERE WITH CONVERSIONS]
To summarize, if TDX uses refcounts to indicate that it is using a page, or to indicate anything else, then we cannot easily split a page on private to shared conversions.
Specifically, consider the case where only the x-th subpage of a huge folio is mapped into Secure-EPTs. When the guest requests to convert some subpage to shared, the huge folio has to be split for core-mm. Core-mm, which will use the shared page, must have split folios to be able to accurately and separately track refcounts for subpages.
During splitting,
guest_memfdwould see refcount of 512 (for 2M page being in the filemap) + 1 (if TDX indicates that the x-th subpage is mapped using a refcount), but would not be able to tell that the 513th refcount belongs to the x-th subpage.guest_memfdcan't split the huge folio unless it knows how to distribute the 513th refcount.One might say
guest_memfdcould clear all the refcounts that TDX is holding on the huge folio by unmapping the entire huge folio from the Secure-EPTs, but unmapping the entire huge folio for TDX means zeroing the contents and requiring guest re-acceptance. Both of these would mess up guest operation.Quoted 7 lines
Hence, guest_memfd's solution is to require that users of
guest_memfdfor private memory trustguest_memfdto maintain the pages around and not take any refcounts.So back to Topic 1, for production kernels, is it okay that TDX does not need to indicate that it is using a page, and can trust
guest_memfdto keep the page around for the VM?I think Yan's concern is not totally invalid. But I don't see a problem if we have a line of sight to adding debug checking as follow on work. That is kind of the path I was trying to nudge.
Quoted 91 lines
Quoted 90 lines
Quoted 8 lines
Topic 2: How to handle unmapping/splitting errors arising from TDX?
Previously I was in favor of having
unmap()return an error (Rick suggested doing a POC, and in a more recent email Rick asked for a diffstat), but Vishal and I talked about this and now I agree having unmapping return an error is not a good approach for these reasons.Ok, let's close this option then.
Quoted 19 lines
1. Unmapping takes a range, and within the range there could be more than one unmapping error. I was previously thinking that
unmap()could return 0 for success and the failed PFN on error. Returning a single PFN on error is okay-ish but if there are more errors it could get complicated.Another error return option could be to return the folio where the unmapping/splitting issue happened, but that would not be sufficiently precise, since a folio could be larger than 4K and we want to track errors as precisely as we can to reduce memory loss due to errors.
2. What I think Yan has been trying to say:
unmap()returning an error is non-standard in the kernel.I think (1) is the dealbreaker here and there's no need to do the plumbing POC and diffstat.
Quoted 8 lines
So I think we're all in support of indicating unmapping/splitting issues without returning anything from
unmap(), and the discussed options area. Refcounts: won't work - mostly discussed in this (sub-)thread [3]. Using refcounts makes it impossible to distinguish between transient refcounts and refcounts due to errors.
b. Page flags: won't work with/can't benefit from HVO.
As above, this was for the purpose of catching bugs, not for guestmemfd to logically depend on it.
Quoted 6 lines
Suggestions still in the running:
c. Folio flags are not precise enough to indicate which page actually had an error, but this could be sufficient if we're willing to just waste the rest of the huge page on unmapping error.
For a scenario of TDX module bug, it seems ok to me.
Quoted 18 lines
d. Folio flags with folio splitting on error. This means that on unmapping/Secure EPT PTE splitting error, we have to split the (larger than 4K) folio to 4K, and then set a flag on the split folio.
The issue I see with this is that splitting pages with HVO applied means doing allocations, and in an error scenario there may not be memory left to split the pages.
e. Some other data structure in
guest_memfd, say, a linked list, and a function likekvm_gmem_add_error_pfn(struct page *page)that would look up theguest_memfdinode from the page and add the page's pfn to the linked list.Everywhere in
guest_memfdthat does unmapping/splitting would then check this linked list to see if the unmapping/splitting succeeded.Quoted 5 lines
Everywhere in
guest_memfdthat allocates pages will also check this linked list to make sure the pages are functional.When
guest_memfdtruncates, if the page being truncated is on the list, retain the refcount on the page and leak that page.I think this is a fine option.
Quoted 10 lines
f. Combination of c and e, something similar to HugeTLB's
folio_set_hugetlb_hwpoison(), which sets a flag AND adds the pages in trouble to a linked list on the folio.g. Like f, but basically treat an unmapping error as hardware poisoning.
I'm kind of inclined towards g, to just treat unmapping errors as HWPOISON and buying into all the HWPOISON handling requirements. What do yall think? Can a TDX unmapping error be considered as memory poisoning?
What does HWPOISON bring over refcounting the page/folio so that it never returns to the page allocator?
Quoted 7 lines
Quoted 3 lines
For Topic 2 (handling TDX unmapping errors), HWPOISON is better than refcounting because refcounting interferes with conversions (see [INTERFERE WITH CONVERSIONS] above).
I don't know if it quite fits. I think it would be better to not pollute the concept if possible.
If there's something we know for sure doesn't fit, and that we're overloading/polluting the concept of HWpoison, then we shouldn't proceed, but otherwise, is it okay to go with HWpoison as a first cut? I replied Yan's email with reasons why I think we should give HWpoison a try, at least for the next RFC.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
We are bugging the TD in these cases.
Bugging the TD does not help to prevent future conversions from being interfered with.
1. Conversions involves unmapping, so we could actually be in a conversion, the unmapping is performed and fails, and then we try to split and enter an infinite loop since private to shared conversions assumes
guest_memfdholds the only refcounts onguest_memfdmemory.
Quoted 8 lines
Quoted 4 lines
2. The conversion ioctl is a
guest_memfdioctl, not a VM ioctl, and so there is no check that the VM is not dead. There shouldn't be any check on the VM, because shareability is a property of the memory and should be changeable independent of the associated VM.Hmm, they are both about unlinking guestmemfd from a VM lifecycle then. Is that a better way to put it?
Unmapping during conversions doesn't take memory away from a VM, it just forces the memory to be re-faulted as shared, so unlinking memory from a VM lifecycle isn't quite accurate, if I understand you correctly.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Ohhh... Is
this about the code to allow gmem fds to be handed to new VMs?
Quoted 8 lines
Quoted 4 lines
Nope, it's not related to linking. The proposed
KVM_LINK_GUEST_MEMFDioctl [4] also doesn't check if the source VM is dead. There shouldn't be any check on the source VM, since the memory is fromguest_memfdand should be independently transferable to a new VM.If a page is mapped in the old TD, and a new TD is started, re-mapping the same page should be prevented somehow, right?
Currently I'm thinking that if we go with HWpoison, the new TD will still get the HWpoison-ed page. The new TD will get the SIGBUS when it next faults the HWpoison-ed page.
Are you thinking that the HWpoison-ed page should be replaced with a non-poisoned page for the new TD to run?
Or are you thinking that
* the handing over should be blocked, or
* mapping itself should be blocked, or
* faulting should be blocked?
If handing over should be blocked, could we perhaps scan for HWpoison when doing the handover and block it there?
I guess I'm trying to do as little as possible during error discovery (hoping to just mark HWpoison), error handling (just unmap from guest page tables, like guest_memfd does now), and defer handling to fault/conversion/perhaps truncation time.
Quoted 3 lines
It really does seem like guestmemfd is the right place to keep the the "stuck page" state. If guestmemfd is not tied to a VM and can be re-used, it should be the one to decide whether they can be mapped again.
Yup, guest_memfd should get to decide.
Quoted 3 lines
Refcounting on error is
about preventing return to the page allocator but that is not the only problem.
guest_memfd, or perhaps the memory_failure() handler for guest_memfd, should prevent this return.
Quoted 5 lines
I do think that these threads have gone on far too long. It's probably about time to move forward with something even if it's just to have something to discuss that doesn't require footnoting so many lore links. So how about we move forward with option e as a next step. Does that sound good Yan?
Please see my reply to Yan, I'm hoping y'all will agree to something between option f/g instead.
Quoted 1 line
Ackerley, thank you very much for pulling together this summary.
Thank you for your reviews and suggestions!
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 9 lines · Ackerley Tng
Quoted 6 lines
I do think that these threads have gone on far too long. It's probably about time to move forward with something even if it's just to have something to discuss that doesn't require footnoting so many lore links. So how about we move
forward with option e as a next step. Does that sound good Yan?Please see my reply to Yan, I'm hoping y'all will agree to something between option f/g instead.
I'm not sure about the HWPoison approach, but I'm not totally against it. My bias is that all the MM concepts are tightly interlinked. If may fit perfectly, but every new use needs to be checked for how fits in with the other MM users of it. Every time I've decided a page flag was the perfect solution to my problem, I got informed otherwise. Let me try to flag Kirill to this discussion. He might have some insights.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 11 lines · Rick P
Quoted 9 lines · Ackerley Tng
Quoted 6 lines
I do think that these threads have gone on far too long. It's probably about time to move forward with something even if it's just to have something to discuss that doesn't require footnoting so many lore links. So how about we move
forward with option e as a next step. Does that sound good Yan?Please see my reply to Yan, I'm hoping y'all will agree to something between option f/g instead.
Quoted 6 lines
I'm not sure about the HWPoison approach, but I'm not totally against it. My bias is that all the MM concepts are tightly interlinked. If may fit perfectly, but every new use needs to be checked for how fits in with the other MM users of it. Every time I've decided a page flag was the perfect solution to my problem, I got informed otherwise. Let me try to flag Kirill to this discussion. He might have some insights.
We chatted with Rick about this.
If I understand correctly, we are discussing the situation where the TDX module failed to return a page to the kernel.
I think it is reasonable to use HWPoison for this case. We cannot guarantee that we will read back whatever we write to the page. TDX module has creative ways to corrupt it.
The memory is no longer functioning as memory. It matches the definition of HWPoison quite closely.
Signature · 1 line
Kiryl Shutsemau / Kirill A. Shutemov
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 19 lines · Kirill
Quoted 13 lines
Quoted 2 lines
Please see my reply to Yan, I'm hoping y'all will agree to something between option f/g instead.
I'm not sure about the HWPoison approach, but I'm not totally against it. My bias is that all the MM concepts are tightly interlinked. If may fit perfectly,
but every new use needs to be checked for how fits in with the other MM users of
it. Every time I've decided a page flag was the perfect solution to my problem,
I got informed otherwise. Let me try to flag Kirill to this discussion. He might
have some insights.We chatted with Rick about this.
If I understand correctly, we are discussing the situation where the TDX module failed to return a page to the kernel.
Quoted 6 lines
I think it is reasonable to use HWPoison for this case. We cannot guarantee that we will read back whatever we write to the page. TDX module has creative ways to corrupt it.
The memory is no longer functioning as memory. It matches the definition of HWPoison quite closely.
ok! Lets go f/g. Unless Yan objects.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 20 lines · Rick P
Quoted 19 lines · Kirill
Quoted 13 lines
Quoted 2 lines
Please see my reply to Yan, I'm hoping y'all will agree to something between option f/g instead.
I'm not sure about the HWPoison approach, but I'm not totally against it. My bias is that all the MM concepts are tightly interlinked. If may fit perfectly,
but every new use needs to be checked for how fits in with the other MM users of
it. Every time I've decided a page flag was the perfect solution to my problem,
I got informed otherwise. Let me try to flag Kirill to this discussion. He might
have some insights.We chatted with Rick about this.
If I understand correctly, we are discussing the situation where the TDX module failed to return a page to the kernel.
Quoted 8 lines
Quoted 6 lines
I think it is reasonable to use HWPoison for this case. We cannot guarantee that we will read back whatever we write to the page. TDX module has creative ways to corrupt it.
The memory is no longer functioning as memory. It matches the definition of HWPoison quite closely.
ok! Lets go f/g. Unless Yan objects.
Follow up as I think about this more: Perhaps we don't need to check for HWpoison (or TDX unmap errors) on conversion.
On a high level, we don't need to check for HWpoison because conversion is about changing memory metadata, as in memory privacy status and struct folio sizes, and not touching memory contents at all. HWpoison means the memory and its contents shouldn't be used.
Specifically for private-to-shared conversions where the TDX unmap error can happen, we will
1. HWpoison the page
2. Bug the TD
This falsely successful conversion means the host (guest_memfd) will think the memory is shared while it may still be mapped in Secure-EPTs.
I think that is okay because the only existing user (TD) stops using that memory, and no future users can use the memory:
1. The TD will be bugged by then. A non-running TD cannot touch memory that had the error on unmapping.
2. The page was not mapped into host page tables (since it was private). Even if it were mapped, it will be unmapped from host page tables (host page table unmaps don't fail). If the host tries to touch the memory, on the next fault, core-mm would notice that the page is poisoned and not fault it in.
By the way, when we "bug the TD", can we assume that ALL vCPUs, not just the one that is did the failed unmap will stop running?
I guess even if the other vCPUs don't stop running, the TDs vCPUs will access the page as shared thinking the conversion succeeded and keep hitting #VEs. If the TD accesses the page as private, it's fine since the page was not unmapped from Secure-EPTs due to the unmap failure and the host cannot write to it (host will see HWpoison on next fault) and so there's no host crash and doesn't defeat the purpose of guest_memfd.
If the guest_memfd with a HWpoisoned page is linked to a new, runnable TD, the new TD would need to fault in the page as private. When it tries to fault in the page to the new TD, it will hit the HWpoison and userspace will get to know about the HWpoison.
Yan, Rick, let me know what you think of this!
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 22 lines · Ackerley Tng
Quoted 20 lines · Rick P
Quoted 19 lines · Kirill
Quoted 13 lines
Quoted 2 lines
Please see my reply to Yan, I'm hoping y'all will agree to something between option f/g instead.
I'm not sure about the HWPoison approach, but I'm not totally against it. My bias is that all the MM concepts are tightly interlinked. If may fit perfectly,
but every new use needs to be checked for how fits in with the other MM users of
it. Every time I've decided a page flag was the perfect solution to my problem,
I got informed otherwise. Let me try to flag Kirill to this discussion. He might
have some insights.We chatted with Rick about this.
If I understand correctly, we are discussing the situation where the TDX module failed to return a page to the kernel.
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
I think it is reasonable to use HWPoison for this case. We cannot guarantee that we will read back whatever we write to the page. TDX module has creative ways to corrupt it.
The memory is no longer functioning as memory. It matches the definition of HWPoison quite closely.
ok! Lets go f/g. Unless Yan objects.
I'm ok with f/g. But I have two implementation specific questions:
1. How to set the HWPoison bit in TDX?
2. Should we set this bit for non-guest-memfd pages (e.g. for S-EPT pages) ?
TDX can't invoke memory_failure() on error of removing guest private pages or S-EPT pages, because holding write mmu_lock is regarded as in atomic context. As there's a mutex in memory_failure(),
"BUG: sleeping function called from invalid context at kernel/locking/mutex.c" will be printed.
If TDX invokes memory_failure_queue() instead, looks guest_memfd can invoke memory_failure_queue_kick() to ensure HWPoison bit is set timely. But which component could invoke memory_failure_queue_kick() for S-EPT pages? KVM?
Quoted 2 lines
Follow up as I think about this more: Perhaps we don't need to check for HWpoison (or TDX unmap errors) on conversion.
Hmm, yes. HWPoision bit is checked in __kvm_gmem_get_pfn() and __do_fault(). Looks we don't need to check it on conversion for the purpose of disallowing shared memory access.
My previous mail was based on another bit and I was not aware of the check of HWPoison in __do_fault().
The conversion will be successful without checking HWPoision during conversion, with error log "MCE: Killing ... due to hardware memory corruption fault at ..." though.
Quoted 20 lines
On a high level, we don't need to check for HWpoison because conversion is about changing memory metadata, as in memory privacy status and struct folio sizes, and not touching memory contents at all. HWpoison means the memory and its contents shouldn't be used.
Specifically for private-to-shared conversions where the TDX unmap error can happen, we will
1. HWpoison the page
2. Bug the TDThis falsely successful conversion means the host (
guest_memfd) will think the memory is shared while it may still be mapped in Secure-EPTs.I think that is okay because the only existing user (TD) stops using that memory, and no future users can use the memory:
1. The TD will be bugged by then. A non-running TD cannot touch memory that had the error on unmapping.
Quoted 8 lines
2. The page was not mapped into host page tables (since it was private). Even if it were mapped, it will be unmapped from host page tables (host page table unmaps don't fail). If the host tries to touch the memory, on the next fault, core-mm would notice that the page is poisoned and not fault it in.
By the way, when we "bug the TD", can we assume that ALL vCPUs, not just the one that is did the failed unmap will stop running?
Right. All the vCPUs will be kicked out of non-root mode after "bug the VM".
Quoted 11 lines
I guess even if the other vCPUs don't stop running, the TDs vCPUs will access the page as shared thinking the conversion succeeded and keep hitting #VEs. If the TD accesses the page as private, it's fine since the page was not unmapped from Secure-EPTs due to the unmap failure and the host cannot write to it (host will see HWpoison on next fault) and so there's no host crash and doesn't defeat the purpose of
guest_memfd.If the
guest_memfdwith a HWpoisoned page is linked to a new, runnable TD, the new TD would need to fault in the page as private. When it tries to fault in the page to the new TD, it will hit the HWpoison and userspace will get to know about the HWpoison.
I'm Ok with just checking HWPosion on the next fault or dequeue of hugetlb.
Quoted 1 line
Yan, Rick, let me know what you think of this!
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 5 lines · Yan Zhao
Quoted 1 line
Quoted 1 line
ok! Lets go f/g. Unless Yan objects.
I'm ok with f/g. But I have two implementation specific questions:
1. How to set the HWPoison bit in TDX?
2. Should we set this bit for non-guest-memfd pages (e.g. for S-EPT pages) ?
Argh, I guess we can keep the existing ref count based approach for the other types of TDX owned pages?
Quoted 11 lines
TDX can't invoke
memory_failure()on error of removing guest private pages or S-EPT pages, because holding writemmu_lockis regarded as in atomic context. As there's a mutex inmemory_failure(),
"BUG: sleeping function called from invalid context atkernel/locking/mutex.c" will be printed.If TDX invokes
memory_failure_queue()instead, looksguest_memfdcan invokememory_failure_queue_kick()to ensure HWPoison bit is set timely. But which component could invokememory_failure_queue_kick()for S-EPT pages? KVM?
Hmm, it only has queue of 10 pages per-cpu. If something goes wrong in the TDX module, I could see exceeding this during a zap operation. At which point, how much have we really handled it?
But, at the risk of derailing the solution when we are close, some reflection has made me question whether this is all misprioritized. We are trying to handle a case where a TDX module bug may return an error when we try to release gmem pages. For that, this solution is feeling way too complex.
If there is a TDX module bug, a simpler way to handle it would be to fix the bug. In the meantime the kernel can take simpler, more drastic efforts to reclaim the memory and ensure system stability.
In the host kexec patches we need to handle a kexec while the TDX module is running. The solution is to simply wbinvd on each pCPU that might have entered the TDX module. After that, barring no new SEAMCALLs that could dirty memory, the pages are free to use by the next kernel. (at least on systems without the partial write errata)
So for this we can do something similar. Have the arch/x86 side of TDX grow a new tdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return a TDX_BUGGY_SHUTDOWN error, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 5 lines · Rick P
Quoted 4 lines · Yan Zhao
Quoted 1 line
Quoted 1 line
ok! Lets go f/g. Unless Yan objects.
I'm ok with f/g. But I have two implementation specific questions:
1. How to set the HWPoison bit in TDX?
I was thinking to set the HWpoison flag based on page type. If regular 4K page, set the flag. If THP page (not (yet) supported by guest_memfd), set the has_hwpoison flag, and if HugeTLB page, call folio_set_hugetlb_hwpoison().
But if we go with Rick's suggestion below, then we don't have to figure this out.
Quoted 5 lines
Quoted 1 line
2. Should we set this bit for non-guest-memfd pages (e.g. for S-EPT pages) ?
Argh, I guess we can keep the existing ref count based approach for the other types of TDX owned pages?
Wait TDX can only use guest_memfd pages, right? Even if TDX can use non-guest_memfd pages, why not also set HWpoison for non-guest_memfd pages?
Either way I guess if we go with Rick's suggestion below, then we don't have to figure the above out.
Quoted 32 lines
Quoted 11 lines
TDX can't invoke
memory_failure()on error of removing guest private pages or S-EPT pages, because holding writemmu_lockis regarded as in atomic context. As there's a mutex inmemory_failure(),
"BUG: sleeping function called from invalid context atkernel/locking/mutex.c" will be printed.If TDX invokes
memory_failure_queue()instead, looksguest_memfdcan invokememory_failure_queue_kick()to ensure HWPoison bit is set timely. But which component could invokememory_failure_queue_kick()for S-EPT pages? KVM?Hmm, it only has queue of 10 pages per-cpu. If something goes wrong in the TDX module, I could see exceeding this during a zap operation. At which point, how much have we really handled it?
But, at the risk of derailing the solution when we are close, some reflection has made me question whether this is all misprioritized. We are trying to handle a case where a TDX module bug may return an error when we try to release gmem pages. For that, this solution is feeling way too complex.
If there is a TDX module bug, a simpler way to handle it would be to fix the bug. In the meantime the kernel can take simpler, more drastic efforts to reclaim the memory and ensure system stability.
In the host kexec patches we need to handle a kexec while the TDX module is running. The solution is to simply wbinvd on each pCPU that might have entered the TDX module. After that, barring no new SEAMCALLs that could dirty memory, the pages are free to use by the next kernel. (at least on systems without the partial write errata)
Quoted 6 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.
Do you mean that on unmap/split failure: there is a way to make 100% sure all memory becomes re-usable by the rest of the host, using tdx_buggy_shutdown(), wbinvd, etc?
If yes, then I'm onboard with this, and if we are 100% sure all memory becomes re-usable by the host after all the extensive cleanup, then we don't need to HWpoison anything.
Quoted 4 lines
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 11 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?
I think so. If we think the error conditions are rare enough that the cost of killing all TDs is acceptable, then we should do a proper POC and give it some scrutiny.
Quoted 4 lines
If yes, then I'm onboard with this, and if we are 100% sure all memory becomes re-usable by the host after all the extensive cleanup, then we don't need to HWpoison anything.
For eventual upstream acceptance, we need to stop and think every time TDX requires special handling in generic code. This is why I wanted to clarify if you guys think the scenario could be in any way considered a generic one. (IOMMU, etc).
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 11 lines · Rick P
Quoted 10 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.
Quoted 3 lines
Quoted 1 line
Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever KVM_BUG_ON() was hit in TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.
So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 7 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous access in shared memory with potential private access from TD or TDX module?
Quoted 13 lines
I think so. If we think the error conditions are rare enough that the cost of killing all TDs is acceptable, then we should do a proper POC and give it some scrutiny.
Quoted 4 lines
If yes, then I'm onboard with this, and if we are 100% sure all memory becomes re-usable by the host after all the extensive cleanup, then we don't need to HWpoison anything.
For eventual upstream acceptance, we need to stop and think every time TDX requires special handling in generic code. This is why I wanted to clarify if you guys think the scenario could be in any way considered a generic one. (IOMMU, etc).
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 26 lines · Yan Zhao
Quoted 11 lines · Rick P
Quoted 10 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.Quoted 3 lines
Quoted 1 line
Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 4 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous access in shared memory with potential private access from TD or TDX module?
If no more seamcalls are allowed and all cpus are made to exit SEAM mode then how can there be potential private access from TD or TDX module?
Quoted 14 lines
Quoted 14 lines
I think so. If we think the error conditions are rare enough that the cost of killing all TDs is acceptable, then we should do a proper POC and give it some scrutiny.
Quoted 4 lines
If yes, then I'm onboard with this, and if we are 100% sure all memory becomes re-usable by the host after all the extensive cleanup, then we don't need to HWpoison anything.
For eventual upstream acceptance, we need to stop and think every time TDX requires special handling in generic code. This is why I wanted to clarify if you guys think the scenario could be in any way considered a generic one. (IOMMU, etc).
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 31 lines · Vishal Annapurve
Quoted 30 lines · Yan Zhao
Quoted 11 lines · Rick P
Quoted 10 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.Quoted 3 lines
Quoted 1 line
Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 7 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?
Quoted 8 lines
Quoted 4 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous access in shared memory with potential private access from TD or TDX module?
If no more seamcalls are allowed and all cpus are made to exit SEAM mode then how can there be potential private access from TD or TDX module?
Not sure. As Kirill said "TDX module has creative ways to corrupt it"
https://lore.kernel.org/all/zlxgzuoqwrbuf54wfqycnuxzxz2yduqtsjinr5uq4ss7iuk2rt@qaaolzwsy6ki/.
Or, could TDX just set a page flag, like what for XEN
/* XEN */
/* Pinned in Xen as a read-only pagetable page. */
PG_pinned = PG_owner_priv_1,e.g.
PG_tdx_firmware_access = PG_owner_priv_1,
Then, guest_memfd checks this flag on every zap and replace it with PG_hwpoison on behalf of TDX?
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 38 lines · Yan Zhao
Quoted 36 lines · Vishal Annapurve
Quoted 34 lines · Yan Zhao
Quoted 11 lines · Rick P
Quoted 10 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.Quoted 3 lines
Quoted 1 line
Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 7 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous access in shared memory with potential private access from TD or TDX module?
Quoted 5 lines
Quoted 3 lines
If no more seamcalls are allowed and all cpus are made to exit SEAM mode then how can there be potential private access from TD or TDX module?
Not sure. As Kirill said "TDX module has creative ways to corrupt it"
https://lore.kernel.org/all/zlxgzuoqwrbuf54wfqycnuxzxz2yduqtsjinr5uq4ss7iuk2rt@qaaolzwsy6ki/.
I would assume that would be true only if TDX module logic is allowed to execute. Otherwise it would be useful to understand these "creative" ways better.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 7 lines · Vishal Annapurve
Quoted 3 lines
Not sure. As Kirill said "TDX module has creative ways to corrupt it"
https://lore.kernel.org/all/zlxgzuoqwrbuf54wfqycnuxzxz2yduqtsjinr5uq4ss7iuk2rt@qaaolzwsy6ki/
.I would assume that would be true only if TDX module logic is allowed to execute. Otherwise it would be useful to understand these "creative" ways better.
The no more seamcalls = no more corruptions is what the host kexec patches are based on.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Yan ZhaoQuoted 49 lines · Yan Zhao
Quoted 31 lines · Vishal Annapurve
Quoted 30 lines · Yan Zhao
Quoted 11 lines · Rick P
Quoted 10 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.Quoted 3 lines
Quoted 1 line
Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 7 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?Quoted 8 lines
Quoted 4 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous access in shared memory with potential private access from TD or TDX module?
If no more seamcalls are allowed and all cpus are made to exit SEAM mode then how can there be potential private access from TD or TDX module?
Not sure. As Kirill said "TDX module has creative ways to corrupt it"
https://lore.kernel.org/all/zlxgzuoqwrbuf54wfqycnuxzxz2yduqtsjinr5uq4ss7iuk2rt@qaaolzwsy6ki/.Or, could TDX just set a page flag, like what for XEN
/* XEN */ /* Pinned in Xen as a read-only pagetable page. */ PG_pinned = PG_owner_priv_1,
Quoted 5 lines
e.g.
PG_tdx_firmware_access=PG_owner_priv_1,Then,
guest_memfdchecks this flag on every zap and replace it withPG_hwpoisonon behalf of TDX?
I think this question probably arose because of a misunderstanding I might have caused. I meant to set the HWpoison flag from the kernel, not from within the TDX module. Please see [1].
In addition, if the TDX module (now referring specifically to the TDX module and not the kernel) sets page flags, that won't work with vmemmap-optimized folios. Setting a page flag on a vmemmap-optimized folio will be setting the flag on a few pages.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 54 lines · Ackerley Tng
Quoted 52 lines · Yan Zhao
Quoted 31 lines · Vishal Annapurve
Quoted 30 lines · Yan Zhao
Quoted 11 lines · Rick P
Quoted 10 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.Quoted 3 lines
Quoted 1 line
Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 7 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?Quoted 8 lines
Quoted 4 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous access in shared memory with potential private access from TD or TDX module?
If no more seamcalls are allowed and all cpus are made to exit SEAM mode then how can there be potential private access from TD or TDX module?
Not sure. As Kirill said "TDX module has creative ways to corrupt it"
https://lore.kernel.org/all/zlxgzuoqwrbuf54wfqycnuxzxz2yduqtsjinr5uq4ss7iuk2rt@qaaolzwsy6ki/.Or, could TDX just set a page flag, like what for XEN
/* XEN */ /* Pinned in Xen as a read-only pagetable page. */ PG_pinned = PG_owner_priv_1,e.g.
PG_tdx_firmware_access=PG_owner_priv_1,
Quoted 6 lines
Quoted 2 lines
Then,
guest_memfdchecks this flag on every zap and replace it withPG_hwpoisonon behalf of TDX?I think this question probably arose because of a misunderstanding I might have caused. I meant to set the HWpoison flag from the kernel, not from within the TDX module. Please see [1].
I understood.
But as Rick pointed out
https://lore.kernel.org/all/[email protected]/,
Manually setting the poison flag in KVM's TDX code (in host kernel) seems risky.
Quoted 2 lines
In addition, if the TDX module (now referring specifically to the TDX module and not the kernel) sets page flags, that won't work with
Marking at per-folio level seems acceptable to me.
Quoted 2 lines
vmemmap-optimized folios. Setting a page flag on a vmemmap-optimized folio will be setting the flag on a few pages.
BTW, I have a concern regarding to the overhead vmemmap-optimization.
In my system,
with hugetlb_free_vmemmap=false, the TD boot time is around 30s; with hugetlb_free_vmemmap=true, the TD boot time is around 1m20s;
Quoted 1 line
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 58 lines · Yan Zhao
Quoted 57 lines · Ackerley Tng
Quoted 54 lines · Yan Zhao
Quoted 31 lines · Vishal Annapurve
Quoted 30 lines · Yan Zhao
Quoted 11 lines · Rick P
Quoted 10 lines · Ackerley Tng
Quoted 9 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a
newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after
that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system
die. Zap/cleanup paths return success in the buggy shutdown case.Quoted 3 lines
Quoted 1 line
Do you mean that on unmap/split failure:
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 7 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures, like the ones that can be fixed with retries, then I think HWPoison is not a good option though.
Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?Quoted 8 lines
Quoted 4 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous access in shared memory with potential private access from TD or TDX module?
If no more seamcalls are allowed and all cpus are made to exit SEAM mode then how can there be potential private access from TD or TDX module?
Not sure. As Kirill said "TDX module has creative ways to corrupt it"
https://lore.kernel.org/all/zlxgzuoqwrbuf54wfqycnuxzxz2yduqtsjinr5uq4ss7iuk2rt@qaaolzwsy6ki/.Or, could TDX just set a page flag, like what for XEN
/* XEN */ /* Pinned in Xen as a read-only pagetable page. */ PG_pinned = PG_owner_priv_1,e.g.
PG_tdx_firmware_access=PG_owner_priv_1,Then,
guest_memfdchecks this flag on every zap and replace it withPG_hwpoisonon behalf of TDX?
Quoted 8 lines
Quoted 3 lines
I think this question probably arose because of a misunderstanding I might have caused. I meant to set the HWpoison flag from the kernel, not from within the TDX module. Please see [1].
I understood.
But as Rick pointed out
https://lore.kernel.org/all/[email protected]/,
Manually setting the poison flag in KVM's TDX code (in host kernel) seems risky.
Will address this in a reply to Rick's email, there's more context there that I'd like to clarify.
Quoted 4 lines
Quoted 2 lines
In addition, if the TDX module (now referring specifically to the TDX module and not the kernel) sets page flags, that won't work with
Marking at per-folio level seems acceptable to me.
Will address this in a reply to Rick's email, there's more context there that I'd like to clarify.
Quoted 9 lines
Quoted 2 lines
vmemmap-optimized folios. Setting a page flag on a vmemmap-optimized folio will be setting the flag on a few pages.
BTW, I have a concern regarding to the overhead vmemmap-optimization.
In my system,
with hugetlb_free_vmemmap=false, the TD boot time is around 30s; with hugetlb_free_vmemmap=true, the TD boot time is around 1m20s;
I'm aware of this, was investigating this for something similar internally. In your system and test, were you working with 1G pages, or 2M pages?
Quoted 1 line
Quoted 1 line
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 11 lines · Ackerley Tng
Quoted 7 lines
Quoted 2 lines
vmemmap-optimized folios. Setting a page flag on a vmemmap-optimized folio will be setting the flag on a few pages.
BTW, I have a concern regarding to the overhead vmemmap-optimization.
In my system,
with hugetlb_free_vmemmap=false, the TD boot time is around 30s; with hugetlb_free_vmemmap=true, the TD boot time is around 1m20s;I'm aware of this, was investigating this for something similar internally. In your system and test, were you working with 1G pages, or 2M pages?
2M pages.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 5 lines · Yan Zhao
Quoted 4 lines · Ackerley Tng
Quoted 4 lines
Quoted 2 lines
vmemmap-optimized folios. Setting a page flag on a vmemmap-optimized folio will be setting the flag on a few pages.
BTW, I have a concern regarding to the overhead vmemmap-optimization.
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
In my system,
with hugetlb_free_vmemmap=false, the TD boot time is around 30s; with hugetlb_free_vmemmap=true, the TD boot time is around 1m20s;I'm aware of this, was investigating this for something similar internally. In your system and test, were you working with 1G pages, or 2M pages?
2M pages.
Thanks for letting me know. I'll look more into this. I'm currently also working on some optimizations, hope this will get addressed.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Yan ZhaoQuoted 8 lines · Yan Zhao
Quoted 2 lines
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in
TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
We were going to call back into guestmemfd for this, right? Not set it inside KVM code.
What about a kvm_gmem_buggy_cleanup() instead of the system wide one. KVM calls it and then proceeds to bug the TD only from the KVM side. It's not as safe for the system, because who knows what a buggy TDX module could do. But TDX module could also be buggy without the kernel catching wind of it.
Having a single callback to basically bug the fd would solve the atomic context issue. Then guestmemfd could dump the entire fd into memory_failure() instead of returning the pages. And developers could respond by fixing the bug.
IMO maintainability needs to be balanced with efforts to minimize the fallout from bugs. In the end a system that is too complex is going to have more bugs anyway.
Quoted 10 lines
Quoted 9 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures,
like the ones that can be fixed with retries, then I think HWPoison is not a good option though.Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?
Quoted 5 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous
access in shared memory with potential private access from TD or TDX module?
With the no more seamcall's approach it should be safe (for the system). This is essentially what we are doing for kexec.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 8 lines · Rick P
Quoted 7 lines · Yan Zhao
Quoted 2 lines
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in
TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.
Quoted 5 lines
Quoted 1 line
So, it could be due to KVM or TDX module bugs, which retries can't help.
We were going to call back into guestmemfd for this, right? Not set it inside KVM code.
Perhaps we had different understandings of f/g :P
I meant that TDX module should directly set the HWpoison flag on the folio (HugeTLB or 4K, guest_memfd or not), not call into guest_memfd.
guest_memfd will then check this flag when necessary, specifically:
* On faults, either into guest or host page tables
* When freeing the page
* guest_memfd will not return HugeTLB pages that are poisoned to HugeTLB and just leak it
* 4K pages will be freed normally, because free_pages_prepare() will check for HWpoison and skip freeing, from __folio_put() -> free_frozen_pages() -> __free_frozen_pages() -> free_pages_prepare()
* I believe guest_memfd doesn't need to check HWpoison on conversions [1]
[1] https://lore.kernel.org/all/[email protected]/
Quoted 9 lines
What about a
kvm_gmem_buggy_cleanup()instead of the system wide one. KVM calls it and then proceeds to bug the TD only from the KVM side. It's not as safe for the system, because who knows what a buggy TDX module could do. But TDX module could also be buggy without the kernel catching wind of it.Having a single callback to basically bug the fd would solve the atomic context issue. Then guestmemfd could dump the entire fd into
memory_failure()instead of returning the pages. And developers could respond by fixing the bug.
This could work too.
I'm in favor of buying into the HWpoison system though, since we're quite sure this is fair use of HWpoison.
Are you saying kvm_gmem_buggy_cleanup() will just set the HWpoison flag on the parts of the folios in trouble?
Quoted 22 lines
IMO maintainability needs to be balanced with efforts to minimize the fallout from bugs. In the end a system that is too complex is going to have more bugs anyway.
Quoted 10 lines
Quoted 9 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures,
like the ones that can be fixed with retries, then I think HWPoison is not a good option though.Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?Quoted 5 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous
access in shared memory with potential private access from TD or TDX module?With the no more seamcall's approach it should be safe (for the system). This is essentially what we are doing for kexec.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 12 lines · Ackerley Tng
Quoted 10 lines · Rick P
Quoted 8 lines · Yan Zhao
Quoted 2 lines
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in
TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.So, it could be due to KVM or TDX module bugs, which retries can't help.
Quoted 9 lines
Quoted 3 lines
We were going to call back into guestmemfd for this, right? Not set it inside KVM code.
Perhaps we had different understandings of f/g :P
I meant that TDX module should directly set the HWpoison flag on the folio (HugeTLB or 4K,
guest_memfdor not), not call intoguest_memfd.
Sorry, correction here, not "TDX module" but the TDX part of KVM within the kernel. Not the TDX module code itself. Sorry for the confusion.
Quoted 54 lines
guest_memfdwill then check this flag when necessary, specifically:* On faults, either into guest or host page tables
* When freeing the page
*guest_memfdwill not return HugeTLB pages that are poisoned to HugeTLB and just leak it
* 4K pages will be freed normally, becausefree_pages_prepare()will check for HWpoison and skip freeing, from__folio_put()->free_frozen_pages()->__free_frozen_pages()->free_pages_prepare()
* I believeguest_memfddoesn't need to check HWpoison on conversions [1][1] https://lore.kernel.org/all/[email protected]/
Quoted 9 lines
What about a
kvm_gmem_buggy_cleanup()instead of the system wide one. KVM calls it and then proceeds to bug the TD only from the KVM side. It's not as safe for the system, because who knows what a buggy TDX module could do. But TDX module could also be buggy without the kernel catching wind of it.Having a single callback to basically bug the fd would solve the atomic context issue. Then guestmemfd could dump the entire fd into
memory_failure()instead of returning the pages. And developers could respond by fixing the bug.This could work too.
I'm in favor of buying into the HWpoison system though, since we're quite sure this is fair use of HWpoison.
Are you saying
kvm_gmem_buggy_cleanup()will just set the HWpoison flag on the parts of the folios in trouble?Quoted 22 lines
IMO maintainability needs to be balanced with efforts to minimize the fallout from bugs. In the end a system that is too complex is going to have more bugs anyway.
Quoted 10 lines
Quoted 9 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures,
like the ones that can be fixed with retries, then I think HWPoison is not a good option though.Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?Quoted 5 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous
access in shared memory with potential private access from TD or TDX module?With the no more seamcall's approach it should be safe (for the system). This is essentially what we are doing for kexec.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Ackerley TngQuoted 1 line · Ackerley Tng
Perhaps we had different understandings of f/g :P
Ah yes, I thought you were saying that guestmemfd would use poison internally via some gmem_buggy_page() or similar. I guess I thought it is more of guestmemfd's job. But as Yan pointed out, we need to handle non gmem page errors too. Currently we leak, but it would be nice to keep the handling symmetrical. Which would be easier if we did it all in TDX code.
Quoted 16 lines
I meant that TDX module should directly set the HWpoison flag on the folio (HugeTLB or 4K,
guest_memfdor not), not call intoguest_memfd.
guest_memfdwill then check this flag when necessary, specifically:* On faults, either into guest or host page tables
* When freeing the page
*guest_memfdwill not return HugeTLB pages that are poisoned to HugeTLB and just leak it
* 4K pages will be freed normally, becausefree_pages_prepare()will check for HWpoison and skip freeing, from__folio_put()->free_frozen_pages()->__free_frozen_pages()->free_pages_prepare()
* I believeguest_memfddoesn't need to check HWpoison on conversions [1]
Quoted 1 line
If a poisoned page continued to be used, it's a bit weird, no? It could take an #MC for another reason from userspace and the handling code would see the page flag is already set. If it doesn't already trip up some MM code somewhere, it might put undue burden on the memory failure code to have to expect repeated poisoning of the same memory.
Quoted 11 lines
Quoted 10 lines
What about a
kvm_gmem_buggy_cleanup()instead of the system wide one. KVM calls it and then proceeds to bug the TD only from the KVM side. It's not as safe for the system, because who knows what a buggy TDX module could do. But TDX module could also be buggy without the kernel catching wind of it.Having a single callback to basically bug the fd would solve the atomic context issue. Then guestmemfd could dump the entire fd into
memory_failure()instead of returning the pages. And developers could respond by fixing the bug.
Quoted 4 lines
This could work too.
I'm in favor of buying into the HWpoison system though, since we're quite sure this is fair use of HWpoison.
Do you mean manually setting the poison flag, or calling into memory_failure(), and friends? If we set them manually, we need to make sure that it does not have side effects on the machine check handler. It seems risky/messy to me. But Kirill didn't seem worried.
Maybe we could bring the poison page flag up to DavidH and see if there is any concern before going down this path too far?
Quoted 3 lines
Are you saying
kvm_gmem_buggy_cleanup()will just set the HWpoison flag on the parts of the folios in trouble?
I was saying kvm_gmem_buggy_cleanup() can set a bool on the fd, similar to VM_BUG_ON() setting vm_dead. After an invalidate, if gmem see this, it needs to assume everything failed, and invalidate everything and poison all guest memory. The point was to have the simplest possible handling for a rare error. Although it's only a proposal. The TDX emergency shutdown option may be simpler still. But killing all TDs is not ideal. So thought we could at least consider other options.
If we have a solution where TDX needs to do something complicated because something of its specialness, it may get NAKed. This is my main concern with the direction of this problem/solution. AFAICT, we are not even sure of a concrete problem, and it appears to be special to TDX. So the complexity budget should be small. It's in sharp contrast to the length of the discussion.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 9 lines · Rick P
Quoted 1 line · Ackerley Tng
Perhaps we had different understandings of f/g :P
Ah yes, I thought you were saying that guestmemfd would use poison internally via some
gmem_buggy_page()or similar. I guess I thought it is more of guestmemfd's job. But as Yan pointed out, we need to handle non gmem page errors too. Currently we leak, but it would be nice to keep the handling symmetrical. Which would be easier if we did it all in TDX code.
I meant to set HWpoison externally from guest_memfd because I feel that it is a separate thing. Unmap failures are similar to discovering a memory error. If setting HWpoison on memory error is external to guest_memfd, setting HWpoison on unmap failure should also be conceptually external to guest_memfd.
After Yan pointed out that non-guest_memfd page errors need to be handled, it aligns with the idea that setting HWpoison is external to guest_memfd.
I agree keeping the handling symmetrical would be best, so in both cases the part of KVM TDX code that sees the unmap failure should directly set HWpoison and not go through guest_memfd.
Quoted 16 lines
Quoted 16 lines
I meant that TDX module should directly set the HWpoison flag on the folio (HugeTLB or 4K,
guest_memfdor not), not call intoguest_memfd.
guest_memfdwill then check this flag when necessary, specifically:* On faults, either into guest or host page tables
* When freeing the page
*guest_memfdwill not return HugeTLB pages that are poisoned to HugeTLB and just leak it
* 4K pages will be freed normally, becausefree_pages_prepare()will check for HWpoison and skip freeing, from__folio_put()->free_frozen_pages()->__free_frozen_pages()->free_pages_prepare()
* I believeguest_memfddoesn't need to check HWpoison on conversions [1]
Quoted 3 lines
Quoted 1 line
If a poisoned page continued to be used, it's a bit weird, no?
Do you mean "continued to be used" in the sense that it is present in a filemap and belongs to a (guest_memfd) inode?
A poisoned page is not faulted in anywhere, and in that sense the page is not "used". In the case of regular poisoning as in a call to memory_failure(), the page is unmapped from the page tables. If that page belongs to guest_memfd, in today's code [2], guest_memfd intentionally does not truncate it from the filemap. For guest_memfd, handling the HWpoison at fault time is by design; keeping it present in the filemap is by design.
In the case of TDX unmap failures leading to HWpoison, the only place it may remain mapped is in the Secure-EPTs. I use "may" because I'm not sure about how badly the unmap failed. But either way, the TD gets bugged, all vCPUs of the TD are stopped, so the HWpoison-ed page is no longer "used".
Quoted 6 lines
It could take an
#MC for another reason from userspace and the handling code would see the page flag is already set. If it doesn't already trip up some MM code somewhere, it might put undue burden on the memory failure code to have to expect repeated poisoning of the same memory.
If it does take another #MC and go to memory_failure(), memory_failure() already checks for the HWpoison flag being set [3]. This is handled by killing the process. There is similar handling for a HugeTLB folio. We're not introducing anything new by using HWpoison; we're buying into the HWpoison framework, which already handles seeing a HWpoison when handling a poison.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
What about a
kvm_gmem_buggy_cleanup()instead of the system wide one. KVM calls it and then proceeds to bug the TD only from the KVM side. It's not as safe for the system, because who knows what a buggy TDX module could do. But TDX module could also be buggy without the kernel catching wind of it.Having a single callback to basically bug the fd would solve the atomic context issue. Then guestmemfd could dump the entire fd into
memory_failure()instead of returning the pages. And developers could respond by fixing the bug.
Quoted 7 lines
Quoted 4 lines
This could work too.
I'm in favor of buying into the HWpoison system though, since we're quite sure this is fair use of HWpoison.
Do you mean manually setting the poison flag, or calling into
memory_failure(), and friends?
I mean manually setting the poison flag.
* If regular 4K page, set the flag.
* If THP page (not (yet) supported by guest_memfd), set the poison flag on the specific subpage causing the error, and in addition set THP'S has_hwpoison flag
* If HugeTLB page, call folio_set_hugetlb_hwpoison() on the subpage.
This is already the process in memory_failure() and perhaps some refactoring could be done.
I think calling memory_failure() would do too much, since in addition to setting the flag, memory_failure() also sometimes does freeing and may kill processes, and triggers the users of the page to further handle the HWpoison.
Quoted 4 lines
If we set them manually, we need to make sure that it does not have side effects on the machine check handler. It seems risky/messy to me. But Kirill didn't seem worried.
I believe the memory_failure() is called from the machine check handler:
DEFINE_IDTENTRY_MCE(exc_machine_check)
-> exc_machine_check_kernel()
-> do_machine_check()
-> kill_me_now() or kill_me_maybe()
-> memory_failure()(I might have quoted just one of the paths and I'll have to look into it more.)
For now, IIUC setting the poison flag is a subset of memory_failure(), which is a subset of what the machine check handler does.
memory_failure() handles an already poisoned page, so I don't see any side effects.
I'm happy that Kirill didn't seem worried :) Rick, let me know if you see any specific risks.
Quoted 3 lines
Maybe we could bring the poison page flag up to DavidH and see if there is any concern before going down this path too far?
I can do that. David's cc-ed on this email, and I hope to get a chance to talk about handling HWpoison (generally, not TDX specifically) at the guest_memfd bi-weekly upstream call on 2025-07-10 so I can bring this up too.
Quoted 6 lines
Quoted 3 lines
Are you saying
kvm_gmem_buggy_cleanup()will just set the HWpoison flag on the parts of the folios in trouble?I was saying
kvm_gmem_buggy_cleanup()can set a bool on the fd, similar toVM_BUG_ON()settingvm_dead.
Setting a bool on the fd is a possible option too. Comparing an inode-level boolean and HWpoison, I still prefer HWpoison because
1. HWpoison gives us more information about which (sub)folio was poisoned. We can think of the bool on the fd as an fd-wide poisoning. If we don't know which subpage has an error, we're forced to leak the entire fd when the inode is released, which could be a huge amount of memory leaked.
2. HWpoison is already checked on faults, so there is no need to add an extra check on a bool
3. For HugeTLB, HWpoison will have to be summarized/itemized on merge/split to handle regular non-TDX related HWpoisons, so no additional code there.
Quoted 3 lines
After an invalidate, if gmem see this, it needs to
assume everything failed, and invalidate everything and poison all guest memory. The point was to have the simplest possible handling for a rare error.
I agree a bool will probably result in fewer lines of code being changed and could be a fair first cut, but I feel like we would very quickly need another patch series to get more granular information and not have to leak an entire fd worth of memory.
Along these lines, Yan seems to prefer setting HWpoison on the entire folio without going into the details of the exact subfolios being poisoned. I think this is a possible in-between solution that doesn't require leaking the entire fd worth of memory, but it still leaks more than just where the actual error happened.
I'm willing to go with just setting HWpoison on the entire large folio as a first cut and leak more memory than necessary (because if we don't know which subpage it is, we are forced to leak everything to be safe).
However, this patch series needs a large page provider in guest_memfd, and will only land either after THP or HugeTLB support lands in guest_memfd.
For now if you're testing on guest_memfd+HugeTLB, folio_set_hugetlb_hwpoison() already exists, why not use it?
Quoted 7 lines
Although
it's only a proposal. The TDX emergency shutdown option may be simpler still. But killing all TDs is not ideal. So thought we could at least consider other options.If we have a solution where TDX needs to do something complicated because something of its specialness, it may get NAKed.
Using HWpoison is generic, since guest_memfd needs to handle HWpoison for regular memory errors anyway. Even if it is not a final solution, it should be good enough, if not for this patch series to merge, at least for the next RFC of this patch series. :)
Quoted 4 lines
This is my main concern with the
direction of this problem/solution. AFAICT, we are not even sure of a concrete problem, and it appears to be special to TDX. So the complexity budget should be small. It's in sharp contrast to the length of the discussion.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 5 lines · Ackerley Tng
Quoted 2 lines
If a poisoned page continued to be used, it's a bit weird, no?
Do you mean "continued to be used" in the sense that it is present in a filemap and belongs to a (
guest_memfd) inode?
I mean anyway where it might get read or written to again.
Quoted 8 lines
A poisoned page is not faulted in anywhere, and in that sense the page is not "used". In the case of regular poisoning as in a call to
memory_failure(), the page is unmapped from the page tables. If that page belongs toguest_memfd, in today's code [2],guest_memfdintentionally does not truncate it from the filemap. Forguest_memfd, handling the HWpoison at fault time is by design; keeping it present in the filemap is by design.
I thought I read that you would allow it to be re-used. I see that the code already checks for poison in the kvm_gmem_get_pfn() path and the mmap() path. So it will just sit in the fd and not be handed out again. I think it's ok. Well, as long as conversion to shared doesn't involve zeroing...?
Quoted 9 lines
In the case of TDX unmap failures leading to HWpoison, the only place it may remain mapped is in the Secure-EPTs. I use "may" because I'm not sure about how badly the unmap failed. But either way, the TD gets bugged, all vCPUs of the TD are stopped, so the HWpoison-ed page is no longer "used".
Yes, I saw that. It looks like special error case treatment for the state we are setting up.
Quoted 10 lines
Quoted 9 lines
It could take an
#MC for another reason from userspace and the handling code would see the page
flag is already set. If it doesn't already trip up some MM code somewhere, it
might put undue burden on the memory failure code to have to expect repeated poisoning of the same memory.
Quoted 6 lines
If it does take another #MC and go to
memory_failure(),memory_failure()already checks for the HWpoison flag being set [3]. This is handled by killing the process. There is similar handling for a HugeTLB folio. We're not introducing anything new by using HWpoison; we're buying into the HWpoison framework, which already handles seeing a HWpoison when handling a poison.
Do you see another user that is setting the poison flag manually like proposed? (i.e. not through memory failure handlers)
Quoted 38 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
What about a
kvm_gmem_buggy_cleanup()instead of the system wide one. KVM calls
it and then proceeds to bug the TD only from the KVM side. It's not as safe for
the system, because who knows what a buggy TDX module could do. But TDX module
could also be buggy without the kernel catching wind of it.Having a single callback to basically bug the fd would solve the atomic context
issue. Then guestmemfd could dump the entire fd intomemory_failure()instead of
returning the pages. And developers could respond by fixing the bug.Quoted 8 lines
Quoted 4 lines
This could work too.
I'm in favor of buying into the HWpoison system though, since we're quite sure this is fair use of HWpoison.
Do you mean manually setting the poison flag, or calling into
memory_failure(),
and friends?I mean manually setting the poison flag.
* If regular 4K page, set the flag.
* If THP page (not (yet) supported byguest_memfd), set the poison flag on the specific subpage causing the error, and in addition set THP'Shas_hwpoison
flag
* If HugeTLB page, callfolio_set_hugetlb_hwpoison()on the subpage.
Quoted 7 lines
This is already the process in
memory_failure()and perhaps some refactoring could be done.I think calling
memory_failure()would do too much, since in addition to setting the flag,memory_failure()also sometimes does freeing and may kill processes, and triggers the users of the page to further handle the HWpoison.
It definitely seem like there is more involved than setting the flag. Which means for our case we should try to understand what we are skipping and how it fits with the rest of the kernel. Is any code the checks for poison assuming that memory_failure() stuff has been done? Stuff like that.
Quoted 8 lines
Quoted 5 lines
If we set them manually, we need to make sure that it does not have side effects on the machine check handler. It seems risky/messy to me. But Kirill didn't seem worried.
I believe the
memory_failure()is called from the machine check handler:
Quoted 8 lines
DEFINE_IDTENTRY_MCE(exc_machine_check)
->exc_machine_check_kernel()
->do_machine_check()
->kill_me_now()orkill_me_maybe()
->memory_failure()(I might have quoted just one of the paths and I'll have to look into it more.)
It looked that way to me too. But it works from other contexts. See MADV_HWPOISON (which is for testing).
Quoted 16 lines
For now, IIUC setting the poison flag is a subset of
memory_failure(), which is a
subset of what the machine check handler does.
memory_failure()handles an already poisoned page, so I don't see any side effects.I'm happy that Kirill didn't seem worried :) Rick, let me know if you see any specific risks.
Quoted 4 lines
Maybe we could bring the poison page flag up to DavidH and see if there is any
concern before going down this path too far?
Quoted 4 lines
I can do that. David's cc-ed on this email, and I hope to get a chance to talk about handling HWpoison (generally, not TDX specifically) at the
guest_memfdbi-weekly upstream call on 2025-07-10 so I can bring this up too.
Ok sounds good. Should we just continue the discussion there? I can try to attend.
Quoted 27 lines
Quoted 7 lines
Quoted 4 lines
Are you saying
kvm_gmem_buggy_cleanup()will just set the HWpoison flag on the parts of the folios in trouble?I was saying
kvm_gmem_buggy_cleanup()can set a bool on the fd, similar toVM_BUG_ON()settingvm_dead.Setting a bool on the fd is a possible option too. Comparing an inode-level boolean and HWpoison, I still prefer HWpoison because
1. HWpoison gives us more information about which (sub)folio was poisoned. We can think of the bool on the fd as an fd-wide poisoning. If we don't know which subpage has an error, we're forced to leak the entire fd when the inode is released, which could be a huge amount of memory leaked.
2. HWpoison is already checked on faults, so there is no need to add an extra check on a bool
3. For HugeTLB, HWpoison will have to be summarized/itemized on merge/split to handle
regular non-TDX related HWpoisons, so no additional code there.Quoted 4 lines
After an invalidate, if gmem see this, it needs to
assume everything failed, and invalidate everything and poison all guest memory.
The point was to have the simplest possible handling for a rare error.
Quoted 4 lines
I agree a bool will probably result in fewer lines of code being changed and could be a fair first cut, but I feel like we would very quickly need another patch series to get more granular information and not have to leak an entire fd worth of memory.
We will only leak an entire VMs worth of memory if there is a bug, the form of which I'm not sure. The kernel doesn't usually have a lot of defensive code to handle for bugs elsewhere. Unless it's to help debugging. But especially for other platform software (bios, etc), it should try to stay out of the job of maintaining code to work around unfixed bugs. And here we are working around *potential bugs*.
So another *possible* solution is to expect TDX module/KVM to work. Kill the TD, return success to the invalidation, and hope that it doesn't do anything to those zombie mappings. It will likely work. Probably much more likely to work then some other warning cases in the kernel. As far as debugging, if strange crashes are observed after a bit splat, it can be a good hint.
Unless Yan has some specific case to worry about that she has been holding on to that makes this error condition a more expected state. That could change things.
Quoted 10 lines
Along these lines, Yan seems to prefer setting HWpoison on the entire folio without going into the details of the exact subfolios being poisoned. I think this is a possible in-between solution that doesn't require leaking the entire fd worth of memory, but it still leaks more than just where the actual error happened.
I'm willing to go with just setting HWpoison on the entire large folio as a first cut and leak more memory than necessary (because if we don't know which subpage it is, we are forced to leak everything to be safe).
Leaking more memory than necessary in a bug case seems totally ok to me.
Quoted 15 lines
However, this patch series needs a large page provider in
guest_memfd, and will only land either after THP or HugeTLB support lands inguest_memfd.For now if you're testing on guest_memfd+HugeTLB,
folio_set_hugetlb_hwpoison()already exists, why not use it?Quoted 7 lines
Although
it's only a proposal. The TDX emergency shutdown option may be simpler still.
But killing all TDs is not ideal. So thought we could at least consider other
options.
Quoted 7 lines
Quoted 2 lines
If we have a solution where TDX needs to do something complicated because something of its specialness, it may get NAKed.
Using HWpoison is generic, since
guest_memfdneeds to handle HWpoison for regular memory errors anyway. Even if it is not a final solution, it should be good enough, if not for this patch series to merge, at least for the next RFC of this patch series. :)
Yes, maybe. If we have a normal, easy, non-imposing solution for handling the error then I won't object.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 9 lines · Rick P
Quoted 5 lines · Ackerley Tng
Quoted 2 lines
If a poisoned page continued to be used, it's a bit weird, no?
Do you mean "continued to be used" in the sense that it is present in a filemap and belongs to a (
guest_memfd) inode?I mean anyway where it might get read or written to again.
Today, when handling memory failures, guest_memfd will unmap the page from the guest and on the next fault, guest_memfd will discover the HWpoison flag and return -EHWPOISON for KVM to handle.
If we go with my proposal, on TDX unmap failure, KVM kills the TD and sets -EHWPOISON on the page.
Hence, the TD will not read/write the poisoned page.
TDX unmap failure implies that the page is private, so it will not be mapped into the host page tables. If it is not in the host page tables, the next access from the host will cause a page fault, and that's when the HWpoison will be discoverd.
Hence, the host will also not read/write the poisoned page. Let me know if you see any way else the poisoned page can be read/written to again.
Quoted 9 lines
Quoted 8 lines
A poisoned page is not faulted in anywhere, and in that sense the page is not "used". In the case of regular poisoning as in a call to
memory_failure(), the page is unmapped from the page tables. If that page belongs toguest_memfd, in today's code [2],guest_memfdintentionally does not truncate it from the filemap. Forguest_memfd, handling the HWpoison at fault time is by design; keeping it present in the filemap is by design.
Quoted 5 lines
I thought I read that you would allow it to be re-used. I see that the code already checks for poison in the
kvm_gmem_get_pfn()path and themmap()path. So it will just sit in the fd and not be handed out again. I think it's ok. Well, as long as conversion to shared doesn't involve zeroing...?
IIUC it is zeroed on unmapping from the guest page tables? Is that done by the TDX module, or by TDX code in KVM? Either way I think both of those should be stopped once the unmap failure is discovered, as part of "killing the TD".
Quoted 30 lines
Quoted 9 lines
In the case of TDX unmap failures leading to HWpoison, the only place it may remain mapped is in the Secure-EPTs. I use "may" because I'm not sure about how badly the unmap failed. But either way, the TD gets bugged, all vCPUs of the TD are stopped, so the HWpoison-ed page is no longer "used".
Yes, I saw that. It looks like special error case treatment for the state we are setting up.
Quoted 10 lines
Quoted 9 lines
It could take an
#MC for another reason from userspace and the handling code would see the page
flag is already set. If it doesn't already trip up some MM code somewhere, it
might put undue burden on the memory failure code to have to expect repeated poisoning of the same memory.Quoted 6 lines
If it does take another #MC and go to
memory_failure(),memory_failure()already checks for the HWpoison flag being set [3]. This is handled by killing the process. There is similar handling for a HugeTLB folio. We're not introducing anything new by using HWpoison; we're buying into the HWpoison framework, which already handles seeing a HWpoison when handling a poison.
Quoted 3 lines
Do you see another user that is setting the poison flag manually like proposed? (i.e. not through memory failure handlers)
As far as I know, this might be the first case of setting the poison flag not through memory failure handlers.
Quoted 46 lines
Quoted 38 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
What about a
kvm_gmem_buggy_cleanup()instead of the system wide one. KVM calls
it and then proceeds to bug the TD only from the KVM side. It's not as safe for
the system, because who knows what a buggy TDX module could do. But TDX module
could also be buggy without the kernel catching wind of it.Having a single callback to basically bug the fd would solve the atomic context
issue. Then guestmemfd could dump the entire fd intomemory_failure()instead of
returning the pages. And developers could respond by fixing the bug.Quoted 8 lines
Quoted 4 lines
This could work too.
I'm in favor of buying into the HWpoison system though, since we're quite sure this is fair use of HWpoison.
Do you mean manually setting the poison flag, or calling into
memory_failure(),
and friends?I mean manually setting the poison flag.
* If regular 4K page, set the flag.
* If THP page (not (yet) supported byguest_memfd), set the poison flag on the specific subpage causing the error, and in addition set THP'Shas_hwpoison
flag
* If HugeTLB page, callfolio_set_hugetlb_hwpoison()on the subpage.Quoted 7 lines
This is already the process in
memory_failure()and perhaps some refactoring could be done.I think calling
memory_failure()would do too much, since in addition to setting the flag,memory_failure()also sometimes does freeing and may kill processes, and triggers the users of the page to further handle the HWpoison.
Quoted 5 lines
It definitely seem like there is more involved than setting the flag. Which means for our case we should try to understand what we are skipping and how it fits with the rest of the kernel. Is any code the checks for poison assuming that
memory_failure()stuff has been done? Stuff like that.
Yup! But I do still think setting HWpoison is good enough to pursue at least for a next RFC patch series, and in the process of testing that series we could learn more. Do you mean that we shouldn't proceed until all of this is verified?
Quoted 36 lines
Quoted 8 lines
Quoted 5 lines
If we set them manually, we need to make sure that it does not have side effects on the machine check handler. It seems risky/messy to me. But Kirill didn't seem worried.
I believe the
memory_failure()is called from the machine check handler:Quoted 8 lines
DEFINE_IDTENTRY_MCE(exc_machine_check)
->exc_machine_check_kernel()
->do_machine_check()
->kill_me_now()orkill_me_maybe()
->memory_failure()(I might have quoted just one of the paths and I'll have to look into it more.)
It looked that way to me too. But it works from other contexts. See
MADV_HWPOISON(which is for testing).Quoted 16 lines
For now, IIUC setting the poison flag is a subset of
memory_failure(), which is a
subset of what the machine check handler does.
memory_failure()handles an already poisoned page, so I don't see any side effects.I'm happy that Kirill didn't seem worried :) Rick, let me know if you see any specific risks.
Quoted 4 lines
Maybe we could bring the poison page flag up to DavidH and see if there is any
concern before going down this path too far?
Quoted 6 lines
Quoted 4 lines
I can do that. David's cc-ed on this email, and I hope to get a chance to talk about handling HWpoison (generally, not TDX specifically) at the
guest_memfdbi-weekly upstream call on 2025-07-10 so I can bring this up too.Ok sounds good. Should we just continue the discussion there?
I think we're at a point where further discussion isn't really useful. Kirill didn't seem worried about using HWpoison, so that's a good sign. I think we can go ahead to use HWpoison for the next RFC of this series and we might learn more through the process of testing it.
Do you prefer to just wait till the next guest_memfd call (now rescheduled to 2025-07-17) before proceeding?
Quoted 3 lines
I can try to
attend.
Sure, thanks! It'll be focused on memory failure handling in general, so TDX will just be another participant.
Quoted 79 lines
Quoted 27 lines
Quoted 7 lines
Quoted 4 lines
Are you saying
kvm_gmem_buggy_cleanup()will just set the HWpoison flag on the parts of the folios in trouble?I was saying
kvm_gmem_buggy_cleanup()can set a bool on the fd, similar toVM_BUG_ON()settingvm_dead.Setting a bool on the fd is a possible option too. Comparing an inode-level boolean and HWpoison, I still prefer HWpoison because
1. HWpoison gives us more information about which (sub)folio was poisoned. We can think of the bool on the fd as an fd-wide poisoning. If we don't know which subpage has an error, we're forced to leak the entire fd when the inode is released, which could be a huge amount of memory leaked.
2. HWpoison is already checked on faults, so there is no need to add an extra check on a bool
3. For HugeTLB, HWpoison will have to be summarized/itemized on merge/split to handle
regular non-TDX related HWpoisons, so no additional code there.Quoted 4 lines
After an invalidate, if gmem see this, it needs to
assume everything failed, and invalidate everything and poison all guest memory.
The point was to have the simplest possible handling for a rare error.Quoted 4 lines
I agree a bool will probably result in fewer lines of code being changed and could be a fair first cut, but I feel like we would very quickly need another patch series to get more granular information and not have to leak an entire fd worth of memory.
We will only leak an entire VMs worth of memory if there is a bug, the form of which I'm not sure. The kernel doesn't usually have a lot of defensive code to handle for bugs elsewhere. Unless it's to help debugging. But especially for other platform software (bios, etc), it should try to stay out of the job of maintaining code to work around unfixed bugs. And here we are working around *potential bugs*.
So another *possible* solution is to expect TDX module/KVM to work. Kill the TD, return success to the invalidation, and hope that it doesn't do anything to those zombie mappings. It will likely work. Probably much more likely to work then some other warning cases in the kernel. As far as debugging, if strange crashes are observed after a bit splat, it can be a good hint.
Unless Yan has some specific case to worry about that she has been holding on to that makes this error condition a more expected state. That could change things.
Quoted 10 lines
Along these lines, Yan seems to prefer setting HWpoison on the entire folio without going into the details of the exact subfolios being poisoned. I think this is a possible in-between solution that doesn't require leaking the entire fd worth of memory, but it still leaks more than just where the actual error happened.
I'm willing to go with just setting HWpoison on the entire large folio as a first cut and leak more memory than necessary (because if we don't know which subpage it is, we are forced to leak everything to be safe).
Leaking more memory than necessary in a bug case seems totally ok to me.
Quoted 18 lines
However, this patch series needs a large page provider in
guest_memfd, and will only land either after THP or HugeTLB support lands inguest_memfd.For now if you're testing on guest_memfd+HugeTLB,
folio_set_hugetlb_hwpoison()already exists, why not use it?Quoted 9 lines
Although
it's only a proposal. The TDX emergency shutdown option may be simpler still.
But killing all TDs is not ideal. So thought we could at least consider other
options.If we have a solution where TDX needs to do something complicated because something of its specialness, it may get NAKed.
Quoted 7 lines
Quoted 4 lines
Using HWpoison is generic, since
guest_memfdneeds to handle HWpoison for regular memory errors anyway. Even if it is not a final solution, it should be good enough, if not for this patch series to merge, at least for the next RFC of this patch series. :)Yes, maybe. If we have a normal, easy, non-imposing solution for handling the error then I won't object.
I believe we have 1 solution now, with 4 options to prevent the memory from being re-used by the host.
1. Kill the TD *and* one of the following to prevent the memory from being re-used by the host:
a. Kill the host
b. HWpoison the memory
c. fd bool, aka inode-wide HWpoison on error
d. Leak the memory directly (not great, will mess up conversions and
guest_memfd inode release)
To push along on this topic, is it okay for us to proceed with HWpoison and find out along the way if it is not easy or imposing?
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 9 lines · Ackerley Tng
Quoted 1 line
Ok sounds good. Should we just continue the discussion there?
I think we're at a point where further discussion isn't really useful. Kirill didn't seem worried about using HWpoison, so that's a good sign. I think we can go ahead to use HWpoison for the next RFC of this series and we might learn more through the process of testing it.
Do you prefer to just wait till the next
guest_memfdcall (now rescheduled to 2025-07-17) before proceeding?
Ah, I missed this and joined the call. :)
At this point, I think I'm strongly in favor of not doing anything here.
Yan and I had a discussion on our internal team chat about this. I'll summarize:
Yan confirmed to me again, that there isn't a specific expected failure here. We are talking about bugs generating the invalidation failure, and leaving the page mapped. But in the case of a bug in a normal VM, a page can also be left mapped too.
What is different here, is we have something (a return code) to check that could catch some of the bugs. But this isn't the only case where a SEACMALL has a spec defined error that we can't handle in a no-fail code path. In those other cases, we handle them by making sure the error won't happen and trigger a VM_BUG_ON() if it does anyway. We can be consistent by just doing the same thing in this case. Implementing it looks like just removing the refcounting in the current code.
And this VM_BUG_ON() will lead to a situation almost like unmapping anyway since the TD can no longer be entered. With future VM shutdown work the pages will not be zeroed at shutdown usually either. So we should not always expect crashes if those pages are returned to the page allocator, even if a bug turns up. Additionally KVM_BUG_ON() will leave a loud warning, allowing us to fix the bug.
But Yan raised a point that might be worth doing something for this case. On the partial write errata platforms (a TDX specific thing), pages that are reclaimed need to be zeroed. So to more cleanly handle this subset of catch-able bugs we are focused on, we could zero the page after the KVM_BUG_ON(). But this still need to be weighed with how much we want to add code to address potential bugs.
So on the benefit side, it is very low to me. The other side is the cost side, which I think is maybe actually a stronger case. We can only make TDX a special case too many times before we will run into upstream problems. Not to lean on Sean here, but he bangs this drum. If we find that we have case where we have to add any specialness for TDX (i.e. making it the only thing that sets the poison bit manually), we should look at changing the TDX arch to address it. I'm not sure what that looks like, but we haven't really tried too hard in that direction yet.
So if TDX has a limited number of "gets to be special" cards, I don't think it is prudent to spend it on something this much of an edge case. So our plan is to rely on the KVM_BUG_ON() for now. And consider TDX arch changes (currently unknown), for how to make the situation cleaner somehow.
Yan, is that your recollection? I guess the other points were that although TDX doesn't need it today, for long term, userspace ABI around invalidations should support failure. But the actual gmem/kvm interface for this can be figured out later. And that external EPT specific TDP MMU code could be tweaked to make things work a little safer around this.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 52 lines · Rick P
Quoted 9 lines · Ackerley Tng
Quoted 1 line
Ok sounds good. Should we just continue the discussion there?
I think we're at a point where further discussion isn't really useful. Kirill didn't seem worried about using HWpoison, so that's a good sign. I think we can go ahead to use HWpoison for the next RFC of this series and we might learn more through the process of testing it.
Do you prefer to just wait till the next
guest_memfdcall (now rescheduled to 2025-07-17) before proceeding?Ah, I missed this and joined the call. :)
At this point, I think I'm strongly in favor of not doing anything here.
Yan and I had a discussion on our internal team chat about this. I'll summarize:
Yan confirmed to me again, that there isn't a specific expected failure here. We are talking about bugs generating the invalidation failure, and leaving the page mapped. But in the case of a bug in a normal VM, a page can also be left mapped too.
What is different here, is we have something (a return code) to check that could catch some of the bugs. But this isn't the only case where a SEACMALL has a spec defined error that we can't handle in a no-fail code path. In those other cases, we handle them by making sure the error won't happen and trigger a
VM_BUG_ON()if it does anyway. We can be consistent by just doing the same thing in this case. Implementing it looks like just removing the refcounting in the current code.And this
VM_BUG_ON()will lead to a situation almost like unmapping anyway since the TD can no longer be entered. With future VM shutdown work the pages will not be zeroed at shutdown usually either. So we should not always expect crashes if those pages are returned to the page allocator, even if a bug turns up. AdditionallyKVM_BUG_ON()will leave a loud warning, allowing us to fix the bug.But Yan raised a point that might be worth doing something for this case. On the partial write errata platforms (a TDX specific thing), pages that are reclaimed need to be zeroed. So to more cleanly handle this subset of catch-able bugs we are focused on, we could zero the page after the
KVM_BUG_ON(). But this still need to be weighed with how much we want to add code to address potential bugs.So on the benefit side, it is very low to me. The other side is the cost side, which I think is maybe actually a stronger case. We can only make TDX a special case too many times before we will run into upstream problems. Not to lean on Sean here, but he bangs this drum. If we find that we have case where we have to add any specialness for TDX (i.e. making it the only thing that sets the poison bit manually), we should look at changing the TDX arch to address it. I'm not sure what that looks like, but we haven't really tried too hard in that direction yet.
Quoted 6 lines
So if TDX has a limited number of "gets to be special" cards, I don't think it is prudent to spend it on something this much of an edge case. So our plan is to rely on the
KVM_BUG_ON()for now. And consider TDX arch changes (currently unknown), for how to make the situation cleaner somehow.Yan, is that your recollection? I guess the other points were that although TDX
I'm ok if KVM_BUG_ON() is considered loud enough to warn about the rare potential corruption, thereby making TDX less special.
Quoted 2 lines
doesn't need it today, for long term, userspace ABI around invalidations should support failure. But the actual gmem/kvm interface for this can be figured out
Could we elaborate what're included in userspace ABI around invalidations?
I'm a bit confused as I think the userspace ABI today supports failure already.
Currently, the unmap API between gmem and KVM does not support failure.
In the future, we hope gmem can check if KVM allows a page to be unmapped before triggering the actual unmap.
Quoted 2 lines
later. And that external EPT specific TDP MMU code could be tweaked to make things work a little safer around this.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 10 lines · Yan Zhao
Quoted 2 lines
Yan, is that your recollection? I guess the other points were that although TDX
I'm ok if
KVM_BUG_ON()is considered loud enough to warn about the rare potential corruption, thereby making TDX less special.Quoted 4 lines
doesn't need it today, for long term, userspace ABI around invalidations should
support failure. But the actual gmem/kvm interface for this can be figured outCould we elaborate what're included in userspace ABI around invalidations?
Let's see what Ackerley says.
Quoted 5 lines
I'm a bit confused as I think the userspace ABI today supports failure already.
Currently, the unmap API between gmem and KVM does not support failure.
Great. I'm just trying to summarize the internal conversations. I think the point was for a future looking user ABI, supporting failure is important. But we don't need the KVM/gmem interface figured out yet.
Quoted 4 lines
In the future, we hope gmem can check if KVM allows a page to be unmapped before
triggering the actual unmap.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 6 lines · Rick P
Quoted 5 lines · Yan Zhao
Quoted 2 lines
Yan, is that your recollection? I guess the other points were that although TDX
I'm ok if
KVM_BUG_ON()is considered loud enough to warn about the rare potential corruption, thereby making TDX less special.
Quoted 8 lines
Quoted 5 lines
Quoted 4 lines
doesn't need it today, for long term, userspace ABI around invalidations should
support failure. But the actual gmem/kvm interface for this can be figured outCould we elaborate what're included in userspace ABI around invalidations?
Let's see what Ackerley says.
There's no specific invalidation command for ioctl but I assume you're referring to the conversion ioctl?
There is a conversion ioctl planned for guest_memfd and the conversion ioctl can return an error. The process of conversion involves invalidating the memory that is to be converted, and for now, guest_memfd assumes unmapping is successful (like Yan says), but that can be changed.
Quoted 10 lines
Quoted 5 lines
I'm a bit confused as I think the userspace ABI today supports failure already.
Currently, the unmap API between gmem and KVM does not support failure.
Great. I'm just trying to summarize the internal conversations. I think the point was for a future looking user ABI, supporting failure is important. But we don't need the KVM/gmem interface figured out yet.
I'm onboard here. So "do nothing" means if there is a TDX unmap failure,
+ KVM_BUG_ON() and hence the TD in question stops running,
+ No more conversions will be possible for this TD since the TD stops running.
+ Other TDs can continue running?
+ No refcounts will be taken for the folio/page where the memory failure happened.
+ No other indication (including HWpoison) anywhere in folio/page to indicate this happened.
+ To round this topic up, do we do anything else as part of "do nothing" that I missed? Is there any record in the TDX module (TDX module itself, not within the kernel)?
I'll probably be okay with an answer like "won't know what will happen", but just checking - what might happen if this page that had an unmap failure gets reused? Suppose the KVM_BUG_ON() is noted but somehow we couldn't get to the machine in time and the machine continues to serve, and the memory is used by
1. Some other non-VM user, something else entirely, say a database?
2. Some new non-TDX VM?
3. Some new TD?
Quoted 4 lines
Quoted 4 lines
In the future, we hope gmem can check if KVM allows a page to be unmapped before
triggering the actual unmap.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 10 lines · Ackerley Tng
I'm onboard here. So "do nothing" means if there is a TDX unmap failure,
+
KVM_BUG_ON()and hence the TD in question stops running,
+ No more conversions will be possible for this TD since the TD stops running.
+ Other TDs can continue running?
+ No refcounts will be taken for the folio/page where the memory failure happened.
+ No other indication (including HWpoison) anywhere in folio/page to indicate this happened.
Yea.
Quoted 3 lines
+ To round this topic up, do we do anything else as part of "do nothing" that I missed? Is there any record in the TDX module (TDX module itself, not within the kernel)?
We should keep this as an option for how to change the TDX module to make this solution safer. For future arch things, we should maybe pursue something that works for TDX connect too, which could be more complicated.
Quoted 2 lines
I'll probably be okay with an answer like "won't know what will happen",
I have not exhaustively looked at that there won't be cascading failures. I think it's reasonable given this is a bug case which we already have a way to catch with a warning.
Quoted 3 lines
but just checking - what might happen if this page that had an unmap failure gets reused?
The TDX module has this thing called the PAMT which records how each physical page is in use. If KVM tries to re-add the page, the SEAMCALL will check PAMT, see it is not in the NDA (Not directly assigned) state, and give an error (TDX_OPERAND_PAGE_METADATA_INCORRECT). This is part of the security enforcement.
Quoted 5 lines
Suppose the
KVM_BUG_ON()is noted but somehow we
couldn't get to the machine in time and the machine continues to serve, and the memory is used by1. Some other non-VM user, something else entirely, say a database?
We are in a "there is a bug" state at this point, which means stability should not be expected to be as good. But it should be optimistically ok to re-use the page as long as the TD is not re-entered, or otherwise actuated via SEAMCALL.
Quoted 1 line
2. Some new non-TDX VM?
Same as (1)
Quoted 1 line
3. Some new TD?
As above, the TDX module should prevent this.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 48 lines · Rick P
Quoted 10 lines · Ackerley Tng
I'm onboard here. So "do nothing" means if there is a TDX unmap failure,
+
KVM_BUG_ON()and hence the TD in question stops running,
+ No more conversions will be possible for this TD since the TD stops running.
+ Other TDs can continue running?
+ No refcounts will be taken for the folio/page where the memory failure happened.
+ No other indication (including HWpoison) anywhere in folio/page to indicate this happened.Yea.
Quoted 3 lines
+ To round this topic up, do we do anything else as part of "do nothing" that I missed? Is there any record in the TDX module (TDX module itself, not within the kernel)?
We should keep this as an option for how to change the TDX module to make this solution safer. For future arch things, we should maybe pursue something that works for TDX connect too, which could be more complicated.
Quoted 2 lines
I'll probably be okay with an answer like "won't know what will happen",
I have not exhaustively looked at that there won't be cascading failures. I think it's reasonable given this is a bug case which we already have a way to catch with a warning.
Quoted 3 lines
but just checking - what might happen if this page that had an unmap failure gets reused?
The TDX module has this thing called the PAMT which records how each physical page is in use. If KVM tries to re-add the page, the SEAMCALL will check PAMT, see it is not in the NDA (Not directly assigned) state, and give an error (
TDX_OPERAND_PAGE_METADATA_INCORRECT). This is part of the security enforcement.Quoted 5 lines
Suppose the
KVM_BUG_ON()is noted but somehow we
couldn't get to the machine in time and the machine continues to serve, and the memory is used by1. Some other non-VM user, something else entirely, say a database?
We are in a "there is a bug" state at this point, which means stability should not be expected to be as good. But it should be optimistically ok to re-use the page as long as the TD is not re-entered, or otherwise actuated via SEAMCALL.
Quoted 7 lines
Quoted 1 line
2. Some new non-TDX VM?
Same as (1)
Quoted 1 line
3. Some new TD?
As above, the TDX module should prevent this.
Thanks for clarifying! SGTM!
Btw, after some more work on handling memory failures for guest_memfd, it now seems like it's better for guest_memfd to not use the HWpoison flag internally either.
So it turns out well that for TDX unmap failures we're aligned on not using HWpoison :)
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Edgecombe, Rick PQuoted 8 lines · Rick P
Quoted 7 lines · Yan Zhao
Quoted 2 lines
Maybe Yan can clarify here. I thought the HWpoison scenario was about TDX module
My thinking is to set HWPoison to private pages whenever
KVM_BUG_ON()was hit in
TDX. i.e., when the page is still mapped in S-EPT but the TD is bugged on and about to tear down.
Quoted 4 lines
Quoted 1 line
So, it could be due to KVM or TDX module bugs, which retries can't help.
We were going to call back into guestmemfd for this, right? Not set it inside KVM code.
Right. I think KVM calling back into guestmemf (via a special folio flag or API) is better than KVM setting HWPoison flag or invoking memory_failure() or its friends.
Quoted 8 lines
What about a
kvm_gmem_buggy_cleanup()instead of the system wide one. KVM calls it and then proceeds to bug the TD only from the KVM side. It's not as safe for the system, because who knows what a buggy TDX module could do. But TDX module could also be buggy without the kernel catching wind of it.Having a single callback to basically bug the fd would solve the atomic context issue. Then guestmemfd could dump the entire fd into
memory_failure()instead of returning the pages. And developers could respond by fixing the bug.
Do you mean dumping the entire memory inside fd? Or just memory with certain folio flags in the fd into memory_failure()?
Quoted 3 lines
IMO maintainability needs to be balanced with efforts to minimize the fallout from bugs. In the end a system that is too complex is going to have more bugs anyway.
Agreed.
To me, having KVM to indicate memory corruption at a folio level (i.e. 2MB or 1GB granularity) is acceptable.
KVM can set a flag (e.g. the flag proposed in
https://lore.kernel.org/all/[email protected]).
guest_memfd can check this flag after every zap or after seeing kvm_gmem_buggy_cleanup(). guest_memfd can choose to report memory_failure() or leak the memory.
But I'm ok if you think dumping and memory_failure() the entire memory inside fd is simpler.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
bugs. Not TDX busy errors, demote failures, etc. If there are "normal" failures,
like the ones that can be fixed with retries, then I think HWPoison is not a good option though.Quoted 3 lines
there is a way to make 100%
sure all memory becomes re-usable by the rest of the host, usingtdx_buggy_shutdown(), wbinvd, etc?
Quoted 8 lines
Quoted 5 lines
Not sure about this approach. When TDX module is buggy and the page is still accessible to guest as private pages, even with no-more SEAMCALLs flag, is it safe enough for guest_memfd/hugetlb to re-assign the page to allow simultaneous
access in shared memory with potential private access from TD or TDX module?With the no more seamcall's approach it should be safe (for the system). This is essentially what we are doing for kexec.
AFAIK, kexec stops devices first by invoking device's shutdown hook. Similarly, "the no more seamcall's approach" should interact with devices to avoid DMAs via private keys.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 6 lines · Yan Zhao
Quoted 3 lines
With the no more seamcall's approach it should be safe (for the system). This is
essentially what we are doing for kexec.AFAIK, kexec stops devices first by invoking device's shutdown hook. Similarly, "the no more seamcall's approach" should interact with devices to avoid DMAs via private keys
This is can't happen today.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Ackerley TngQuoted 4 lines · Ackerley Tng
Quoted 2 lines · Rick P
Quoted 1 line · Yan Zhao
Quoted 1 line
Quoted 1 line
ok! Lets go f/g. Unless Yan objects.
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
I'm ok with f/g. But I have two implementation specific questions:
1. How to set the HWPoison bit in TDX?
I was thinking to set the HWpoison flag based on page type. If regular 4K page, set the flag. If THP page (not (yet) supported by
guest_memfd), set thehas_hwpoisonflag, and if HugeTLB page, callfolio_set_hugetlb_hwpoison().
Could you elaborate on how to call folio_set_hugetlb_hwpoison()?
Quoted 5 lines
But if we go with Rick's suggestion below, then we don't have to figure this out.
Quoted 2 lines
Quoted 1 line
2. Should we set this bit for non-guest-memfd pages (e.g. for S-EPT pages) ?
Quoted 7 lines
Quoted 3 lines
Argh, I guess we can keep the existing ref count based approach for the other types of TDX owned pages?
Wait TDX can only use
guest_memfdpages, right? Even if TDX can use non-guest_memfd pages, why not also set HWpoison for non-guest_memfd pages?
As in https://lore.kernel.org/all/[email protected]/,
I don't find a proper interface for TDX to set HWpoison bit on non-guset_memfd pages.
Neither memory_failure() nor memory_failure_queue() seem fit.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 7 lines · Yan Zhao
Quoted 6 lines · Ackerley Tng
Quoted 4 lines · Rick P
Quoted 3 lines · Yan Zhao
Quoted 1 line
Quoted 1 line
ok! Lets go f/g. Unless Yan objects.
I'm ok with f/g. But I have two implementation specific questions:
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
1. How to set the HWPoison bit in TDX?
I was thinking to set the HWpoison flag based on page type. If regular 4K page, set the flag. If THP page (not (yet) supported by
guest_memfd), set thehas_hwpoisonflag, and if HugeTLB page, callfolio_set_hugetlb_hwpoison().Could you elaborate on how to call
folio_set_hugetlb_hwpoison()?
Sorry I meant "in TDX" as in the part of the kernel that performs the unmap. I'm assuming something like
int ret = tdx_do_unmap(page)
if (ret)
set_hwpoison_based_on_folio_type(page_folio(page))
And set_hwpoison_based_on_folio_type() would have to be written to know how to set the HWpoison flag based on type of the folio.
I think I might have used the wrong terminology elsewhere. Sorry about that. I don't mean to call folio_set_hugetlb_hwpoison() from within the TDX module. I meant to set HWpoison in the kernel, based on return value to the kernel from the TDX module.
Quoted 17 lines
Quoted 5 lines
But if we go with Rick's suggestion below, then we don't have to figure this out.
Quoted 2 lines
Quoted 1 line
2. Should we set this bit for non-guest-memfd pages (e.g. for S-EPT pages) ?
Quoted 7 lines
Quoted 3 lines
Argh, I guess we can keep the existing ref count based approach for the other types of TDX owned pages?
Wait TDX can only use
guest_memfdpages, right? Even if TDX can use non-guest_memfd pages, why not also set HWpoison for non-guest_memfd pages?As in https://lore.kernel.org/all/[email protected]/,
I don't find a proper interface for TDX to set HWpoison bit on non-guset_memfd pages.Neither
memory_failure()normemory_failure_queue()seem fit.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 25 lines · Ackerley Tng
Quoted 7 lines · Yan Zhao
Quoted 6 lines · Ackerley Tng
Quoted 4 lines · Rick P
Quoted 3 lines · Yan Zhao
Quoted 1 line
Quoted 1 line
ok! Lets go f/g. Unless Yan objects.
I'm ok with f/g. But I have two implementation specific questions:
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
1. How to set the HWPoison bit in TDX?
I was thinking to set the HWpoison flag based on page type. If regular 4K page, set the flag. If THP page (not (yet) supported by
guest_memfd), set thehas_hwpoisonflag, and if HugeTLB page, callfolio_set_hugetlb_hwpoison().Could you elaborate on how to call
folio_set_hugetlb_hwpoison()?Sorry I meant "in TDX" as in the part of the kernel that performs the unmap. I'm assuming something like
int ret =
tdx_do_unmap(page)
if (ret)
set_hwpoison_based_on_folio_type(page_folio(page))
Quoted 25 lines
And
set_hwpoison_based_on_folio_type()would have to be written to know how to set the HWpoison flag based on type of the folio.I think I might have used the wrong terminology elsewhere. Sorry about that. I don't mean to call
folio_set_hugetlb_hwpoison()from within the TDX module. I meant to set HWpoison in the kernel, based on return value to the kernel from the TDX module.Quoted 17 lines
Quoted 5 lines
But if we go with Rick's suggestion below, then we don't have to figure this out.
Quoted 2 lines
Quoted 1 line
2. Should we set this bit for non-guest-memfd pages (e.g. for S-EPT pages) ?
Quoted 7 lines
Quoted 3 lines
Argh, I guess we can keep the existing ref count based approach for the other types of TDX owned pages?
Wait TDX can only use
guest_memfdpages, right? Even if TDX can use non-guest_memfd pages, why not also set HWpoison for non-guest_memfd pages?As in https://lore.kernel.org/all/[email protected]/,
I don't find a proper interface for TDX to set HWpoison bit on non-guset_memfd pages.Neither
memory_failure()normemory_failure_queue()seem fit.
Missed out a response on this.
Vishal explained to me that non-guest_memfd pages can be used by TDX for the TDX module itself.
For those, I think it's still okay to set HWpoison, because the kernel page freeing process will leak HWpoison-ed pages. free_pages_prepare() will check for HWpoison and skip freeing:
__folio_put() ->
free_frozen_pages() ->
__free_frozen_pages() ->
free_pages_prepare()Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Edgecombe, Rick PQuoted 43 lines · Rick P
Quoted 5 lines · Yan Zhao
Quoted 1 line
Quoted 1 line
ok! Lets go f/g. Unless Yan objects.
I'm ok with f/g. But I have two implementation specific questions:
1. How to set the HWPoison bit in TDX?
2. Should we set this bit for non-guest-memfd pages (e.g. for S-EPT pages) ?Argh, I guess we can keep the existing ref count based approach for the other types of TDX owned pages?
Quoted 11 lines
TDX can't invoke
memory_failure()on error of removing guest private pages or S-EPT pages, because holding writemmu_lockis regarded as in atomic context. As there's a mutex inmemory_failure(),
"BUG: sleeping function called from invalid context atkernel/locking/mutex.c" will be printed.If TDX invokes
memory_failure_queue()instead, looksguest_memfdcan invokememory_failure_queue_kick()to ensure HWPoison bit is set timely. But which component could invokememory_failure_queue_kick()for S-EPT pages? KVM?Hmm, it only has queue of 10 pages per-cpu. If something goes wrong in the TDX module, I could see exceeding this during a zap operation. At which point, how much have we really handled it?
But, at the risk of derailing the solution when we are close, some reflection has made me question whether this is all misprioritized. We are trying to handle a case where a TDX module bug may return an error when we try to release gmem pages. For that, this solution is feeling way too complex.
If there is a TDX module bug, a simpler way to handle it would be to fix the bug. In the meantime the kernel can take simpler, more drastic efforts to reclaim the memory and ensure system stability.
In the host kexec patches we need to handle a kexec while the TDX module is running. The solution is to simply wbinvd on each pCPU that might have entered the TDX module. After that, barring no new SEAMCALLs that could dirty memory, the pages are free to use by the next kernel. (at least on systems without the partial write errata)
Quoted 5 lines
So for this we can do something similar. Have the
arch/x86side of TDX grow a newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.
This approach makes sense to me.
Quoted 6 lines
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Edgecombe, Rick PQuoted 5 lines · Rick P
So for this we can do something similar. Have the
arch/x86side of TDX grow a newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.
All TDs in the system die could be too severe for unmap errors due to KVM bugs.
Quoted 4 lines
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
I didn't mention TDX connect previously to avoid introducing unnecessary complexity.
For TDX connect, S-EPT is used for private mappings in IOMMU. Unmap could therefore fail due to pages being pinned for DMA.
So, my thinking was that if that happens, KVM could set a special flag to folios pinned for private DMA.
Then guest_memfd could check the special flag before allowing private-to-shared conversion, or punch hole.guest_memfd could check this special flag and choose to poison or leak the folio.
Otherwise, if we choose tdx_buggy_shutdown() to "do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool", DMAs may still have access to the private pages mapped in S-EPT.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 8 lines · Yan Zhao
Quoted 5 lines · Rick P
So for this we can do something similar. Have the
arch/x86side of TDX grow a newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.All TDs in the system die could be too severe for unmap errors due to KVM bugs.
At this point, I don't see a way to quantify how bad a KVM bug can get unless you have explicit ideas about the severity. We should work on minimizing KVM side bugs too and assuming it would be a rare occurrence I think it's ok to take this intrusive measure.
Quoted 10 lines
Quoted 5 lines
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
I didn't mention TDX connect previously to avoid introducing unnecessary complexity.
For TDX connect, S-EPT is used for private mappings in IOMMU. Unmap could therefore fail due to pages being pinned for DMA.
We are discussing this scenario already[1], where the host will not pin the pages used by secure DMA for the same reasons why we can't have KVM pin the guest_memfd pages mapped in SEPT. Is there some other kind of pinning you are referring to?
If there is an ordering in which pages should be unmapped e.g. first in secure IOMMU and then KVM SEPT, then we can ensure the right ordering between invalidation callbacks from guest_memfd.
[1] https://lore.kernel.org/lkml/CAGtprH_qh8sEY3s-JucW3n1Wvoq7jdVZDDokvG5HzPf0HV2=pg@mail.gmail.com/#t
Quoted 4 lines
So, my thinking was that if that happens, KVM could set a special flag to folios pinned for private DMA.
Quoted 8 lines
Then
guest_memfdcould check the special flag before allowing private-to-shared conversion, or punch hole.guest_memfdcould check this special flag and choose to poison or leak the folio.Otherwise, if we choose
tdx_buggy_shutdown()to "do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool", DMAs may still have access to the private pages mapped in S-EPT.
guest_memfd will have to ensure that pages are unmapped from secure IOMMU pagetables before allowing them to be used by the host.
If secure IOMMU pagetables unmapping fails, I would assume it fails in the similar category of rare "KVM/TDX module/IOMMUFD" bug and I think it makes sense to do the same tdx_buggy_shutdown() with such failures as well.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 51 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Quoted 5 lines · Rick P
So for this we can do something similar. Have the
arch/x86side of TDX grow a newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.All TDs in the system die could be too severe for unmap errors due to KVM bugs.
At this point, I don't see a way to quantify how bad a KVM bug can get unless you have explicit ideas about the severity. We should work on minimizing KVM side bugs too and assuming it would be a rare occurrence I think it's ok to take this intrusive measure.
Quoted 10 lines
Quoted 5 lines
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
I didn't mention TDX connect previously to avoid introducing unnecessary complexity.
For TDX connect, S-EPT is used for private mappings in IOMMU. Unmap could therefore fail due to pages being pinned for DMA.
We are discussing this scenario already[1], where the host will not pin the pages used by secure DMA for the same reasons why we can't have KVM pin the
guest_memfdpages mapped in SEPT. Is there some other kind of pinning you are referring to?If there is an ordering in which pages should be unmapped e.g. first in secure IOMMU and then KVM SEPT, then we can ensure the right ordering between invalidation callbacks from
guest_memfd.[1] https://lore.kernel.org/lkml/CAGtprH_qh8sEY3s-JucW3n1Wvoq7jdVZDDokvG5HzPf0HV2=pg@mail.gmail.com/#t
Quoted 4 lines
So, my thinking was that if that happens, KVM could set a special flag to folios pinned for private DMA.
Quoted 8 lines
Then
guest_memfdcould check the special flag before allowing private-to-shared conversion, or punch hole.guest_memfdcould check this special flag and choose to poison or leak the folio.Otherwise, if we choose
tdx_buggy_shutdown()to "do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool", DMAs may still have access to the private pages mapped in S-EPT.
Quoted 7 lines
guest_memfdwill have to ensure that pages are unmapped from secure IOMMU pagetables before allowing them to be used by the host.If secure IOMMU pagetables unmapping fails, I would assume it fails in the similar category of rare "KVM/TDX module/IOMMUFD" bug and I think it makes sense to do the same
tdx_buggy_shutdown()with such failures as well.
In addition we will need a way to fail all further Secure IOMMU table walks or some way to stop the active secure DMA by unbinding all the TDIs. Maybe such scenarios warrant a BUG_ON() if recovery is not possible as possibly any or all of the KVM/IOMMUFD/TDX module can't be trusted for reliable functionality anymore.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
Quoted 8 lines · Vishal Annapurve
Quoted 7 lines
guest_memfdwill have to ensure that pages are unmapped from secure IOMMU pagetables before allowing them to be used by the host.If secure IOMMU pagetables unmapping fails, I would assume it fails in the similar category of rare "KVM/TDX module/IOMMUFD" bug and I think it makes sense to do the same
tdx_buggy_shutdown()with such failures as well.
Quoted 5 lines
In addition we will need a way to fail all further Secure IOMMU table walks or some way to stop the active secure DMA by unbinding all the TDIs. Maybe such scenarios warrant a
BUG_ON()if recovery is not possible as possibly any or all of the KVM/IOMMUFD/TDX module can't be trusted for reliable functionality anymore.
I mentioned this on another thread. Normal kernel BUG_ON()'s need extreme justification. As long as the system might survive, they shouldn't be used.
Edgecombe, Rick P
maintainerRe: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Vishal AnnapurveQuoted 8 lines · Vishal Annapurve
Quoted 7 lines · Yan Zhao
Quoted 5 lines · Rick P
So for this we can do something similar. Have the
arch/x86side of TDX grow a newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.
Quoted 6 lines
Quoted 1 line
All TDs in the system die could be too severe for unmap errors due to KVM bugs.
At this point, I don't see a way to quantify how bad a KVM bug can get unless you have explicit ideas about the severity. We should work on minimizing KVM side bugs too and assuming it would be a rare occurrence I think it's ok to take this intrusive measure.
Yes, it does seem on the line of "too severe". But keeping a list of pages to release in a non-atomic context seems to complex for an error case that (still not 100% clear) is theoretical.
In the argument of it's too severe, it's close to a BUG_ON() for the TDX side of the kernel. But on the argument of it's not too severe, the system remains stable.
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
I didn't mention TDX connect previously to avoid introducing unnecessary complexity.
Quoted 7 lines
Quoted 2 lines
For TDX connect, S-EPT is used for private mappings in IOMMU. Unmap could therefore fail due to pages being pinned for DMA.
We are discussing this scenario already[1], where the host will not pin the pages used by secure DMA for the same reasons why we can't have KVM pin the
guest_memfdpages mapped in SEPT. Is there some other kind of pinning you are referring to?
I'm wondering about the "something went wrong and we can't invalidate" pattern. Like the device refuses to cooperate.
Quoted 6 lines
If there is an ordering in which pages should be unmapped e.g. first in secure IOMMU and then KVM SEPT, then we can ensure the right ordering between invalidation callbacks from
guest_memfd.[1] https://lore.kernel.org/lkml/CAGtprH_qh8sEY3s-JucW3n1Wvoq7jdVZDDokvG5HzPf0HV2=pg@mail.gmail.com/#t
The general gist seems to be that guestmemfd should be the nerve center of these decisions, and it should be given enough information to make the decisions to invalidate only when success is guaranteed. Makes sense.
In this case we can't know the condition ahead of time. It is a TDX-only problem? If it is, then we need to make TDX behave more like the others. Or have simple to maintain cop-outs like this.
Quoted 14 lines
Quoted 5 lines
So, my thinking was that if that happens, KVM could set a special flag to folios pinned for private DMA.
Quoted 8 lines
Then
guest_memfdcould check the special flag before allowing private-to-shared conversion, or punch hole.guest_memfdcould check this special flag and choose to poison or leak the folio.Otherwise, if we choose
tdx_buggy_shutdown()to "do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool", DMAs may still have access to the private pages mapped in S-EPT.
Quoted 7 lines
guest_memfdwill have to ensure that pages are unmapped from secure IOMMU pagetables before allowing them to be used by the host.If secure IOMMU pagetables unmapping fails, I would assume it fails in the similar category of rare "KVM/TDX module/IOMMUFD" bug and I think it makes sense to do the same
tdx_buggy_shutdown()with such failures as well.
It's too hypothetical to reason about. IMO, we need to know about specific similar patterns to justify a more complex fine grained poisoning approach.
Re: [RFC PATCH 08/21] KVM: TDX: Increase/decrease folio ref for huge pages
In reply to Vishal AnnapurveQuoted 26 lines · Vishal Annapurve
Quoted 8 lines · Yan Zhao
Quoted 5 lines · Rick P
So for this we can do something similar. Have the
arch/x86side of TDX grow a newtdx_buggy_shutdown(). Have it do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool. Then any SEAMCALLs after that will return aTDX_BUGGY_SHUTDOWNerror, or similar. All TDs in the system die. Zap/cleanup paths return success in the buggy shutdown case.All TDs in the system die could be too severe for unmap errors due to KVM bugs.
At this point, I don't see a way to quantify how bad a KVM bug can get unless you have explicit ideas about the severity. We should work on minimizing KVM side bugs too and assuming it would be a rare occurrence I think it's ok to take this intrusive measure.
Quoted 10 lines
Quoted 5 lines
Does it fit? Or, can you guys argue that the failures here are actually non- special cases that are worth more complex recovery? I remember we talked about IOMMU patterns that are similar, but it seems like the remaining cases under discussion are about TDX bugs.
I didn't mention TDX connect previously to avoid introducing unnecessary complexity.
For TDX connect, S-EPT is used for private mappings in IOMMU. Unmap could therefore fail due to pages being pinned for DMA.
Quoted 8 lines
We are discussing this scenario already[1], where the host will not pin the pages used by secure DMA for the same reasons why we can't have KVM pin the
guest_memfdpages mapped in SEPT. Is there some other kind of pinning you are referring to?If there is an ordering in which pages should be unmapped e.g. first in secure IOMMU and then KVM SEPT, then we can ensure the right ordering between invalidation callbacks from
guest_memfd.
It's pinning from a different perspective.
Please check
https://lore.kernel.org/all/[email protected].
Quoted 15 lines
[1] https://lore.kernel.org/lkml/CAGtprH_qh8sEY3s-JucW3n1Wvoq7jdVZDDokvG5HzPf0HV2=pg@mail.gmail.com/#t
Quoted 4 lines
So, my thinking was that if that happens, KVM could set a special flag to folios pinned for private DMA.
Quoted 8 lines
Then
guest_memfdcould check the special flag before allowing private-to-shared conversion, or punch hole.guest_memfdcould check this special flag and choose to poison or leak the folio.Otherwise, if we choose
tdx_buggy_shutdown()to "do an all-cpu IPI to kick CPUs out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool", DMAs may still have access to the private pages mapped in S-EPT.
Quoted 7 lines
guest_memfdwill have to ensure that pages are unmapped from secure IOMMU pagetables before allowing them to be used by the host.If secure IOMMU pagetables unmapping fails, I would assume it fails in the similar category of rare "KVM/TDX module/IOMMUFD" bug and I think it makes sense to do the same
tdx_buggy_shutdown()with such failures as well.
tdx_buggy_shutdown() should then
do an all-cpu IPI to kick CPU out of SEAMMODE, wbivnd, and set a "no more seamcalls" bool" and informing IOMMUF/VFIO to stop devices.
BTW, is the "no more seamcall" set by KVM at the per-VM level? If it's per-VM, other TDs could still entering SEAMMODE. So, potential corruption is still possible.
Besides, with "no more seamcalls" upon unmapping failure of a GFN, how to reclaim other pages which might succeed otherwise?
This approach seems very complex.