Skip to contentSkip to conversation

KVM: TDX huge page support for private memory

RFC PATCH Superseded Yan Zhao 9 participants 129 messages Aug 7, 2025 – Jan 6

Subsystems (15)
x86 KVM PowerPC MIPS +11 more subsystems

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Yan Zhao

author
#0 lore raw

v2 cover KVM: TDX huge page support for private memory

Hi,

This is RFC v2 series to support huge pages in TDX, currently focusing on supporting 2MB huge pages only. (code available at [2]).

The main goal for RFC v2 is to have the TDX side feedback ready.

Tip folks, there are some SEAMCALL wrapper changes in this series, but we need to have some discussion on the KVM side to figure out what it needs still. Please feel free to ignore it for now."

4 opens in RFC v1
====
Open 1: How to pass guest's ACCEPT level info.

The TDX guest can affect KVM's mapping level of a fault due to the current implementation of the guest ACCEPT operation in the TDX module. If KVM maps at a higher level than the guest's ACCEPT level, repeated faults are generated, prompting KVM to split the huge mapping to match the ACCEPT level. To prevent these repeated faults and subsequent splitting (at least for Linux TDs), it's efficient to pass the guest's ACCEPT level info to the KVM MMU core to restrict the mapping level of a fault.

Prior to RFC v1, this info was conveyed by modifying each fault's error code [10][11].
In RFC v1, the info was passed through the hook private_max_mapping_level, checking a per-vCPU violation_request_level inside TDX [12].

In RFC v2, Sean and Rick suggested a global way [17] (vs a fault-by-fault per-vCPU way) to specify the info. This involves setting the disallow_lpage's GUEST_INHIBIT bit in a slot's lpage_infos above the specified guest ACCEPT level, indicating that the guest wants to prevent a specific GFN from being mapped at those higher levels (Patches 13 and 14). This approach helps prevent potential subtle bugs that could arise from different ACCEPT levels specified by different vCPUs.

Open 2: Do we need to support huge pages on non-Linux TDs and the split in
fault path.

The current TDX module requires tdh_mem_range_block() to be invoked before each tdh_mem_page_demote(). It leads to the complexity of handling BUSY errors under a shared mmu_lock, since if a BUSY error is returned from tdh_mem_page_demote(), TDX must call tdh_mem_range_unblock() (which may also return a BUSY error) before passing the error to the KVM MMU core.

For Linux TDs, guest ACCEPT operations usually occur before KVM populates the mappings, except when KVM pre-faults certain mappings. Therefore, to avoid splitting in the fault path for Linux TDs, RFC v1 required prefetch faults to be mapped at 4KB and KVM's mapping level to honor the guest's ACCEPT level.

For non-Linux TDs, where guest ACCEPT operations may occur after KVM has populated mappings, KVM's mapping level has no way to honor the not-yet-specified guest's ACCEPT level. If the guest later accepts at a level smaller than KVM's mapping level, splitting in the fault path may be necessary. In RFC v1, splitting in the fault path was ignored, with the expectation that the guest would never accept at a lower level than KVM's mapping level. This expectation was later proven incorrect. Forcing a KVM's mapping to be at 4KB before the guest's ACCEPT level is available would disable huge pages for non-Linux TDs as the TD's later ACCEPT operations at a higher level would return error TDX_PAGE_SIZE_MISMATCH to the guest.

In RFC v2, the fault path splitting issue is addressed by having TDX's EPT violation handler:
(1) explicitly split previous huge mapping under write mmu_lock,
(2) set GUEST_INHIBIT bit in a slot's lpage_info above the specified guest ACCEPT level,
before invoking KVM MMU core's kvm_mmu_page_fault().

This mechanism allows support of huge pages for non-Linux TDs and also removes the 4KB restriction on pre-fault mappings for Linux TDs in RFC v1.

It's a TODO to support split in the fault path once new TDX module supporting blockless tdh_mem_page_demote() is available. After that, step (1) would be unnecessary and step (2) could be optimized to be under shared mmu_lock to enhance performance.

Open 3: How to handle the TDX_INTERRUPTED_RESTARTABLE error from
tdh_mem_page_demote().

Currently, the TDX module returns TDX_INTERRUPTED_RESTARTABLE when it receives interrupts (including NMIs) during the SEAMCALL tdh_mem_page_demote(). However, a new TDX module is in planning, which will disable the interrupt checking (including for NMIs) during SEAMCALL tdh_mem_page_demote() for basic TDX (w/ or w/o DPAMT).

As a result, the loop for endlessly retrying TDX_INTERRUPTED_RESTARTABLE has been removed in RFC v2.

The retry logic remains in the code tree [2] for testing purpose with current TDX module.

Open 4: How to handle unmap failures.

To enable guest_memfd to support in-place conversion between shared and private memory [5], TDX is required not to hold refcount of the private pages allocated from guest_memfd. (see details in patch 6).

Without TDX holding extra folio refcount for memory that are still mapped in S-EPT, the guest_memfd may have freed a page while it's still mapped in the S-EPT, since the current KVM unmap operation does not return errors to its callers. (For TDX, the error could arise from failures in SEAMCALLs tdh_mem_page_remove(), tdh_phymem_page_wbinvd_hkid(), tdh_phymem_page_reclaim()).

Several approaches have been explored. (Refer to the links in patch 6). Due to the complexity and given that S-EPT zapping failure is currently only possible due to bugs in the KVM or TDX module, which are very rare in a production kernel, this RFC v2 adopts a straightforward approach -- simply avoiding holding the page reference count in TDX and generating a KVM_BUG_ON() on S-EPT unmap failure without propagating the error back to guest_memfd or notifying guest_memfd through any out-of-band method. To be robust against bugs, the user can enable panic_on_warn as normal.

Main changes from RFC v1
====
- Switched from using 2MB THP series [4] to HugeTLB based in-place conversion series v2 [5] as the huge page allocator. Patches 17 is the updated implementation in guest_memfd for punch_hole and private-to-shared conversion.

- Rebased onto DPAMT series [7] and pulled patches from [7.1] for DPAMT related changes in TDX huge pages. (Patches 19-23).

- Addressed the 4 opens in RFC v1.

- Updated tdh_mem_page_aug(), tdh_phymem_page_wbinvd_hkid(), tdh_phymem_page_reclaim(), introduced tdx_clear_folio() to support "folios" rather than "pages".

- Renamed kvm_split_boundary_leafs() to kvm_split_cross_boundary_leafs() and made it to leverage the existing tdp_mmu_split_huge_pages_root(). The API is now available to both direct and mirror roots and also usable under shared mmu_lock.

- To address the lock issues present in the gmem series [4][5], this RFC v2 is now based on the RFC patch "KVM: TDX: Decouple TDX init mem region from kvm_gmem_populate()" [9].
(Will rebase to Sean's proposed fixes ([13]-[16]) later when the formal patches are available. It's not expected to introduce more than minimal change to the implementation of the TDX huge page series.)

Design
====
1. guest_memfd
Although the TDX huge pages series uses guest_memfd as the huge page allocator, it does not depend on the implementation specifics of guest_memfd. Therefore, aside from the guest_memfd-specific code in patch 17, which calls kvm_split_cross_boundary_leafs() to split private mappings for hole punching and private-to-shared conversion, the TDX huge pages series can function with either 2MB THP guest_memfd [4] (by omitting patch 17) or HugeTLB-based in-place conversion guest_memfd [5].

2. Page size during TD build time
It's forced to 4KB in the last patch (patch 23), because
- tdh_mem_page_add() only adds private pages at 4KB.
- No need to allow merging or splitting during TD build time.

3. Page size during TD runtime
kvm_tdp_mmu_page_fault() can create mappings up to the 2MB level, as determined by TDX's private_max_mapping_level hook. This function can be triggered by host pre-faults, guest ACCEPT operations, or guest GPA accesses.

KVM's mapping size can be influenced by the guest's ACCEPT size. When faults that carry the guest's ACCEPT level occur, the guest ACCEPT level info is passed to the KVM MMU core by setting the GUEST_INHIBIT bit in a slot's lpage_info above the guest ACCEPT level. Consequently, the KVM MMU core will map up to the guest's ACCEPT level.

Currently, the GUEST_INHIBIT bit is set in a one-off manner under a write mmu_lock. Once set, it will not be unset, and the GFN will not be allowed to be mapped at higher levels, even if the mappings are zapped and re-accepted by the guest at higher levels later. This approach simplifies the code and has been tested to have a minor impact on the huge page map count with a typical Linux TD. Future optimizations may include setting the GUEST_INHIBIT bit under a shared mmu_lock or unsetting the GUEST_INHIBIT bit upon zapping.

4. Page splitting (page demotion)
Page splitting can occur due to (1) private-to-shared conversion,
(2) hole punching in guest_memfd, or (3) a guest's ACCEPT operation at a lower level than KVM's created mapping.

With the current TDX module, splitting huge mappings in S-EPT requires executing tdh_mem_range_block(), tdh_mem_track(), kicking off vCPUs, and tdh_mem_page_demote() in sequence. If DPAMT is involved, tdh_phymem_pamt_add() is also required (see bullet 5).

Currently, only scenario (3) may trigger page splitting under a shared mmu_lock. However, to avoid the complexity of handling BUSY errors when tdh_mem_page_demote() depends on tdh_mem_range_block(), TDX page splitting under a shared mmu_lock is not supported in this series (patch 11 directly returns -EOPNOTSUPP). This is also prevented by triggering the splitting under an exclusive mmu_lock on a fault caused by (3) and setting the GUEST_INHIBIT bit in a slot's lpage_info above the guest ACCEPT level. Supporting split in the fault path is a TODO for when a new TDX module that supports blockless tdh_mem_page_demote() becomes available.

With an exclusive kvm->mmu_lock, TDX_OPERAND_BUSY is handled similarly to removing a private page, i.e., by kicking off all vCPUs and retrying, which should succeed on the second attempt.

TDX_INTERRUPTED_RESTARTABLE will not be returned from tdh_mem_page_demote() for basic TDX in the new TDX module that is currently in planning. Therefore, this error is not handled in this RFC v2.

5. Dynamic PAMT
Currently, DPAMT's involvement with TDX huge pages is limited to page splitting.

This part of design depends on the outcome of the DPAMT series [7] design discussions.

The call stack based on the current design is as follows:

kvm_split_cross_boundary_leafs
kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs
tdp_mmu_split_cross_boundary_leafs
tdp_mmu_split_huge_pages_root
1.1 tdp_mmu_alloc_sp_for_split
1.2 topup_mirror_caches <-------------------------------
2.tdp_mmu_split_huge_page |
tdp_mmu_link_sp |
tdp_mmu_iter_set_spte |
tdp_mmu_set_spte |
split_external_spt |
kvm_x86_call(split_external_spt) |
tdx_sept_split_private_spt |
3.1 BLOCK, TRACK |
3.2 PAMT_ADD for the adding page --|
table page |
3.3 alloc pamt pages for the 2MB --
guest page and DEMOTE

1.2 allocates PAMT pages in the per-VM KVM MMU memory cache for both
the adding page table page and the 2MB guest page to be demoted. (Patch 21).

3.2 and 3.3 retrieve the PAMT pages from the per-VM KVM MMU memory
cache and perform tdh_phymem_pamt_add() and tdh_mem_page_demote(). Holding of the exclusive mmu_lock after 1.2 and re-checking the count of free pages ensure that 3.2 and 3.3 can safely acquire enough pages in an atomic context.

6. Page merging (page promotion)
Promotion is disallowed (in patch 7), because
- The current TDX module requires all 4KB leafs to be either all PENDING or all ACCEPTED before a successful promotion to 2MB. This requirement prevents successful page merging after partially converting a 2MB range from private to shared and then back to private, which is the primary scenario necessitating page promotion.
- tdh_mem_page_promote() depends on tdh_mem_range_block() in the current TDX module. Consequently, handling BUSY errors is complex, as page merging typically occurs in the fault path under a shared mmu_lock.

Patches Layout
====
- Patches 01-05: Update SEAMCALL wrappers or page clearing helper.

- Patch 06: Drop holding gmem page refcount in TDX to facilitate guest_memfd in-place conversion [5].

- Patch 07: Disallow page merging for TDX.

- Patches 08-11: Enable KVM MMU core to propagate splitting requests to TDX and provide the corresponding implementation in TDX.

- Patch 12: Introduce a higher level API to split pages.

- Patches 13-14: Split and inhibit huge pages in EPT violation handler.

- Patches 15-17: Split huge pages on page conversion/punch hole.

- Patches 18-22: Dynamic PAMT related changes for TDX huge pages.

- Patch 23: Turn on 2MB huge pages.

Testing
===
The kernel code is available at [2].

The code base includes
- kvm-x86-next-2025.07.21 (with shutdown optimization [8])
- guest_memfd code [6]
- DPAMT v2 [7]
- TDX selftests out of [6].

TDX huge pages can be tested with the KVM selftest tdx_vm_huge_page in [2] or with the matching QEMU [3].

The 2MB mapping count can be checked via "/sys/kernel/debug/kvm/pages_2m".

Huge pages has been verified to work with DPAMT and shutdown optimization on TDX module version TDX_1.5.20.00.887, though further testing is needed (e.g., the mmu stress test).

Thanks
Yan

[1] RFC v1: https://lore.kernel.org/all/[email protected]
[2] Kernel code: https://github.com/intel/tdx/tree/huge_page_v2
[3] QEMU code: https://github.com/intel-staging/qemu-tdx/tree/gmem-tdx-hugepage-v2
[4] 2M THP: https://lore.kernel.org/all/[email protected]
[5] 1G: https://lore.kernel.org/all/[email protected]
[6] https://github.com/googleprodkernel/linux-cc/commits/wip-tdx-gmem-conversions-hugetlb-2mept-v2
[7] DPAMT: https://lore.kernel.org/all/[email protected]
[7.1] git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge
[8] shutdown: https://lore.kernel.org/all/[email protected]
[9] decouple: https://lore.kernel.org/all/[email protected]
[10] https://lore.kernel.org/all/4d61104bff388a081ff8f6ae4ac71e05a13e53c3.1708933624.git.isaku.yamahata@intel.com/
[11]https://lore.kernel.org/all/3d2a6bfb033ee1b51f7b875360bd295376c32b54.1708933624.git.isaku.yamahata@intel.com/
[12] https://lore.kernel.org/all/[email protected]
[13] https://lore.kernel.org/all/[email protected]/
[14] https://lore.kernel.org/all/[email protected]/
[15] https://lore.kernel.org/lkml/[email protected]
[16] https://lore.kernel.org/all/[email protected]
[17] https://lore.kernel.org/all/[email protected]

Edgecombe, Rick P (1):
KVM: x86/mmu: Disallow page merging (huge page adjustment) for mirror
root

Isaku Yamahata (1):
KVM: x86/tdp_mmu: Alloc external_spt page for mirror page table
splitting

Kirill A. Shutemov (5):
x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC
is set
KVM: TDX: Pass down pfn to split_external_spt()
KVM: TDX: Handle Dynamic PAMT in tdh_mem_page_demote()
KVM: TDX: Preallocate PAMT pages to be used in split path
KVM: TDX: Handle Dynamic PAMT on page split

Xiaoyao Li (1):
x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Yan Zhao (15):
x86/tdx: Enhance tdh_mem_page_aug() to support huge pages x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge
pages
KVM: TDX: Introduce tdx_clear_folio() to clear huge pages x86/tdx: Enhance tdh_phymem_page_reclaim() to support huge pages
KVM: TDX: Do not hold page refcount on private guest pages
KVM: x86/tdp_mmu: Add split_external_spt hook called during write
mmu_lock
KVM: TDX: Enable huge page splitting under write kvm->mmu_lock
KVM: x86: Reject splitting huge pages under shared mmu_lock for mirror
root
KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()
KVM: x86: Introduce hugepage_set_guest_inhibit()
KVM: TDX: Split and inhibit huge mappings if a VMExit carries level
info
KVM: Change the return type of gfn_handler_t() from bool to int
KVM: x86: Split cross-boundary mirror leafs for
KVM_SET_MEMORY_ATTRIBUTES
KVM: guest_memfd: Split for punch hole and private-to-shared
conversion
KVM: TDX: Turn on PG_LEVEL_2M after TD is RUNNABLE

 arch/arm64/kvm/mmu.c               |   8 +-
 arch/loongarch/kvm/mmu.c           |   8 +-
 arch/mips/kvm/mmu.c                |   6 +-
 arch/powerpc/kvm/book3s.c          |   4 +-
 arch/powerpc/kvm/e500_mmu_host.c   |   8 +-
 arch/riscv/kvm/mmu.c               |  12 +-
 arch/x86/include/asm/kvm-x86-ops.h |   1 +
 arch/x86/include/asm/kvm_host.h    |   7 +
 arch/x86/include/asm/tdx.h         |  18 ++-
 arch/x86/kvm/mmu.h                 |   3 +
 arch/x86/kvm/mmu/mmu.c             |  85 ++++++++---
 arch/x86/kvm/mmu/tdp_mmu.c         | 208 ++++++++++++++++++++++---
 arch/x86/kvm/mmu/tdp_mmu.h         |   3 +
 arch/x86/kvm/vmx/tdx.c             | 238 ++++++++++++++++++++++-------
 arch/x86/kvm/vmx/tdx_arch.h        |   3 +
 arch/x86/virt/vmx/tdx/tdx.c        | 102 ++++++++++---
 arch/x86/virt/vmx/tdx/tdx.h        |   1 +
 include/linux/kvm_host.h           |  14 +-
 virt/kvm/guest_memfd.c             | 142 ++++++++++-------
 virt/kvm/kvm_main.c                |  37 +++--
 20 files changed, 691 insertions(+), 217 deletions(-)
Signature · 1 line
2.43.2
author
#1 lore raw

RFC PATCH v2 1/23 x86/tdx: Enhance tdh_mem_page_aug() to support huge pages

Enhance the SEAMCALL wrapper tdh_mem_page_aug() to support huge pages.

The SEAMCALL TDH_MEM_PAGE_AUG currently supports adding physical memory to the S-EPT up to 2MB in size.

While keeping the "level" parameter in the tdh_mem_page_aug() wrapper to allow callers to specify the physical memory size, introduce the parameters "folio" and "start_idx" to specify the physical memory starting from the page at "start_idx" within the "folio". The specified physical memory must be fully contained within a single folio.

Invoke tdx_clflush_page() for each 4KB segment of the physical memory being added. tdx_clflush_page() performs CLFLUSH operations on certain TDX-capable platforms, or conservatively on all TDX-capable platforms, to prevent dirty cache lines from writing back later and corrupting TD memory.

---
RFC v2:
- Refine patch log. (Rick)
- Removed the level checking. (Kirill, Chao Gao)
- Use "folio", and "start_idx" rather than "page".
- Return TDX_OPERAND_INVALID if the specified physical memory is not contained within a single folio.
- Use PTE_SHIFT to replace the 9 in "1 << (level * 9)" (Kirill)
- Use C99-style definition of variables inside a loop. (Nikolay Borisov)

RFC v1:
- Rebased to new tdh_mem_page_aug() with "struct page *" as param.
- Check folio, folio_page_idx.

---
 arch/x86/include/asm/tdx.h  |  3 ++-
 arch/x86/kvm/vmx/tdx.c      |  4 +++-
 arch/x86/virt/vmx/tdx/tdx.c | 14 +++++++++++---
 3 files changed, 16 insertions(+), 5 deletions(-)
arch/x86/include/asm/tdx.h +2 -1
index 48d579092590..f968b736871a 100644
@@ -171,7 +171,8 @@ u64 tdh_mng_addcx(struct tdx_td *td, struct page *tdcs_page);
 u64 tdh_mem_page_add(struct tdx_td *td, u64 gpa, struct page *page, struct page *source, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_mem_sept_add(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page);
-u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2);
+u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct folio *folio,
+		     unsigned long start_idx, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_mem_range_block(struct tdx_td *td, u64 gpa, int level, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_mng_key_config(struct tdx_td *td);
 u64 tdh_mng_create(struct tdx_td *td, u16 hkid);
arch/x86/kvm/vmx/tdx.c +3 -1
index ed67f842b6ec..0a2b183899d8 100644
@@ -1593,11 +1593,13 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
 {
 	int tdx_level = pg_level_to_tdx_sept_level(level);
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	struct folio *folio = page_folio(page);
 	gpa_t gpa = gfn_to_gpa(gfn);
 	u64 entry, level_state;
 	u64 err;
 
-	err = tdh_mem_page_aug(&kvm_tdx->td, gpa, tdx_level, page, &entry, &level_state);
+	err = tdh_mem_page_aug(&kvm_tdx->td, gpa, tdx_level, folio,
+			       folio_page_idx(folio, page), &entry, &level_state);
 	if (unlikely(tdx_operand_busy(err))) {
 		tdx_unpin(kvm, page);
 		return -EBUSY;
arch/x86/virt/vmx/tdx/tdx.c +11 -3
index e411cf878547..580f14f64822 100644
@@ -1730,16 +1730,24 @@ u64 tdh_vp_addcx(struct tdx_vp *vp, struct page *tdcx_page)
 }
 EXPORT_SYMBOL_GPL(tdh_vp_addcx);
 
-u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct page *page, u64 *ext_err1, u64 *ext_err2)
+u64 tdh_mem_page_aug(struct tdx_td *td, u64 gpa, int level, struct folio *folio,
+		     unsigned long start_idx, u64 *ext_err1, u64 *ext_err2)
 {
+	struct page *start = folio_page(folio, start_idx);
+	unsigned long npages = 1 << (level * PTE_SHIFT);
 	struct tdx_module_args args = {
 		.rcx = gpa | level,
 		.rdx = tdx_tdr_pa(td),
-		.r8 = page_to_phys(page),
+		.r8 = page_to_phys(start),
 	};
 	u64 ret;
 
-	tdx_clflush_page(page);
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;
+
+	for (int i = 0; i < npages; i++)
+		tdx_clflush_page(nth_page(start, i));
+
 	ret = seamcall_ret(TDH_MEM_PAGE_AUG, &args);
 
 	*ext_err1 = args.rcx;
Signature · 1 line
2.43.2
author
#2 lore raw

RFC PATCH v2 2/23 x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

In reply to Yan Zhao

From: Xiaoyao Li <[email protected]>

Introduce SEAMCALL wrapper tdh_mem_page_demote() to invoke the SEAMCALL TDH_MEM_PAGE_DEMOTE, which demotes a huge leaf entry to a non-leaf entry in the S-EPT.

SEAMCALL TDH_MEM_PAGE_DEMOTE supports the demotion of 2MB or 1GB huge leaf entries.

The "gpa" and "level" parameters enable the SEAMCALL TDH_MEM_PAGE_DEMOTE to walk the S-EPT for the huge leaf entry that needs to be demoted.

The "page" parameter specifies a 4KB page that will be used in the demotion operation to be added as a page table page in the S-EPT.

Invoke tdx_clflush_page() on the 4KB page being added as a page table page. This function performs CLFLUSH operations on certain TDX-capable platforms, or conservatively on all TDX-capable platforms, to prevent dirty cache lines from writing back later and corrupting TD memory.

tdh_mem_page_demote() may fail. Callers can check function return value and retrieve extended error info from the function output parameters "ext_err1" and "ext_err2". e.g., due to S-EPT walk error or arriving interrupts.

The TDX module has many internal locks. To avoid staying in SEAM mode for too long, SEAMCALLs return a BUSY error code to the kernel instead of spinning on the locks. Depending on the specific SEAMCALL, the caller may need to handle this error in specific ways (e.g., retry). Therefore, return the SEAMCALL error code directly to the caller without attempting to handle it in the core kernel.

Do not handle TDX_INTERRUPTED_RESTARTABLE because SEAMCALL TDH_MEM_PAGE_DEMOTE does not check interrupts (including NMIs) for basic TDX (with or without Dynamic PAMT).

---
RFC v2:
- Refine the patch log (Rick).
- Do not handle TDX_INTERRUPTED_RESTARTABLE as the new TDX modules in planning do not check interrupts for basic TDX.

RFC v1:
- Rebased and split patch. Updated patch log.

---
 arch/x86/include/asm/tdx.h  |  2 ++
 arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++
 arch/x86/virt/vmx/tdx/tdx.h |  1 +
 3 files changed, 23 insertions(+)
arch/x86/include/asm/tdx.h +2 -0
index f968b736871a..d2cf48e273d5 100644
@@ -178,6 +178,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);
 u64 tdh_mng_create(struct tdx_td *td, u16 hkid);
 u64 tdh_vp_create(struct tdx_td *td, struct tdx_vp *vp);
 u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data);
+u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
+			u64 *ext_err1, u64 *ext_err2);
 u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_mr_finalize(struct tdx_td *td);
 u64 tdh_vp_flush(struct tdx_vp *vp);
arch/x86/virt/vmx/tdx/tdx.c +20 -0
index 580f14f64822..d941f083f741 100644
@@ -1825,6 +1825,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data)
 }
 EXPORT_SYMBOL_GPL(tdh_mng_rd);
 
+u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
+			u64 *ext_err1, u64 *ext_err2)
+{
+	struct tdx_module_args args = {
+		.rcx = gpa | level,
+		.rdx = tdx_tdr_pa(td),
+		.r8 = page_to_phys(page),
+	};
+	u64 ret;
+
+	tdx_clflush_page(page);
+	ret = seamcall_ret(TDH_MEM_PAGE_DEMOTE, &args);
+
+	*ext_err1 = args.rcx;
+	*ext_err2 = args.rdx;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tdh_mem_page_demote);
+
 u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2)
 {
 	struct tdx_module_args args = {
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 096c78a1d438..a6c0fa53ece9 100644
@@ -24,6 +24,7 @@
 #define TDH_MNG_KEY_CONFIG		8
 #define TDH_MNG_CREATE			9
 #define TDH_MNG_RD			11
+#define TDH_MEM_PAGE_DEMOTE		15
 #define TDH_MR_EXTEND			16
 #define TDH_MR_FINALIZE			17
 #define TDH_VP_FLUSH			18
Signature · 1 line
2.43.2
#3 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 31 lines · Yan Zhao

From: Xiaoyao Li <[email protected]>

Introduce SEAMCALL wrapper tdh_mem_page_demote() to invoke the SEAMCALL TDH_MEM_PAGE_DEMOTE, which demotes a huge leaf entry to a non-leaf entry in the S-EPT.

SEAMCALL TDH_MEM_PAGE_DEMOTE supports the demotion of 2MB or 1GB huge leaf entries.

The "gpa" and "level" parameters enable the SEAMCALL TDH_MEM_PAGE_DEMOTE to walk the S-EPT for the huge leaf entry that needs to be demoted.

The "page" parameter specifies a 4KB page that will be used in the demotion operation to be added as a page table page in the S-EPT.

Invoke tdx_clflush_page() on the 4KB page being added as a page table page. This function performs CLFLUSH operations on certain TDX-capable platforms, or conservatively on all TDX-capable platforms, to prevent dirty cache lines from writing back later and corrupting TD memory.

tdh_mem_page_demote() may fail. Callers can check function return value and retrieve extended error info from the function output parameters "ext_err1" and "ext_err2". e.g., due to S-EPT walk error or arriving interrupts.

The TDX module has many internal locks. To avoid staying in SEAM mode for too long, SEAMCALLs return a BUSY error code to the kernel instead of spinning on the locks. Depending on the specific SEAMCALL, the caller may need to handle this error in specific ways (e.g., retry). Therefore, return the SEAMCALL error code directly to the caller without attempting to handle it in the core kernel.

Quoted 3 lines

Do not handle TDX_INTERRUPTED_RESTARTABLE because SEAMCALL TDH_MEM_PAGE_DEMOTE does not check interrupts (including NMIs) for basic TDX (with or without Dynamic PAMT).

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Quoted 37 lines

---
RFC v2:
- Refine the patch log (Rick).
- Do not handle TDX_INTERRUPTED_RESTARTABLE as the new TDX modules in planning do not check interrupts for basic TDX.

RFC v1:
- Rebased and split patch. Updated patch log.

---
  arch/x86/include/asm/tdx.h  |  2 ++
  arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++
  arch/x86/virt/vmx/tdx/tdx.h |  1 +
  3 files changed, 23 insertions(+)
arch/x86/include/asm/tdx.h +2 -0
index f968b736871a..d2cf48e273d5 100644
@@ -178,6 +178,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);
  u64 tdh_mng_create(struct tdx_td *td, u16 hkid);
  u64 tdh_vp_create(struct tdx_td *td, struct tdx_vp *vp);
  u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data);
+u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
+			u64 *ext_err1, u64 *ext_err2);
  u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2);
  u64 tdh_mr_finalize(struct tdx_td *td);
  u64 tdh_vp_flush(struct tdx_vp *vp);
arch/x86/virt/vmx/tdx/tdx.c +0 -0
index 580f14f64822..d941f083f741 100644

arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1828

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
@@ -1825,6 +1825,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data)
  }
  EXPORT_SYMBOL_GPL(tdh_mng_rd);
 
+u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,

Nit: Is it better to use a var name that clearly tell that the page is used as a table page?

Quoted 34 lines
arch/x86/virt/vmx/tdx/tdx.c +19 -0
+			u64 *ext_err1, u64 *ext_err2)
+{
+	struct tdx_module_args args = {
+		.rcx = gpa | level,
+		.rdx = tdx_tdr_pa(td),
+		.r8 = page_to_phys(page),
+	};
+	u64 ret;
+
+	tdx_clflush_page(page);
+	ret = seamcall_ret(TDH_MEM_PAGE_DEMOTE, &args);
+
+	*ext_err1 = args.rcx;
+	*ext_err2 = args.rdx;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tdh_mem_page_demote);
+
  u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2)
  {
  	struct tdx_module_args args = {
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 096c78a1d438..a6c0fa53ece9 100644
@@ -24,6 +24,7 @@
  #define TDH_MNG_KEY_CONFIG		8
  #define TDH_MNG_CREATE			9
  #define TDH_MNG_RD			11
+#define TDH_MEM_PAGE_DEMOTE		15
  #define TDH_MR_EXTEND			16
  #define TDH_MR_FINALIZE			17
  #define TDH_VP_FLUSH			18
author
#4 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 34 lines · Binbin Wu
Quoted 31 lines · Yan Zhao

From: Xiaoyao Li <[email protected]>

Introduce SEAMCALL wrapper tdh_mem_page_demote() to invoke the SEAMCALL TDH_MEM_PAGE_DEMOTE, which demotes a huge leaf entry to a non-leaf entry in the S-EPT.

SEAMCALL TDH_MEM_PAGE_DEMOTE supports the demotion of 2MB or 1GB huge leaf entries.

The "gpa" and "level" parameters enable the SEAMCALL TDH_MEM_PAGE_DEMOTE to walk the S-EPT for the huge leaf entry that needs to be demoted.

The "page" parameter specifies a 4KB page that will be used in the demotion operation to be added as a page table page in the S-EPT.

Invoke tdx_clflush_page() on the 4KB page being added as a page table page. This function performs CLFLUSH operations on certain TDX-capable platforms, or conservatively on all TDX-capable platforms, to prevent dirty cache lines from writing back later and corrupting TD memory.

tdh_mem_page_demote() may fail. Callers can check function return value and retrieve extended error info from the function output parameters "ext_err1" and "ext_err2". e.g., due to S-EPT walk error or arriving interrupts.

The TDX module has many internal locks. To avoid staying in SEAM mode for too long, SEAMCALLs return a BUSY error code to the kernel instead of spinning on the locks. Depending on the specific SEAMCALL, the caller may need to handle this error in specific ways (e.g., retry). Therefore, return the SEAMCALL error code directly to the caller without attempting to handle it in the core kernel.

Quoted 8 lines
Quoted 3 lines

Do not handle TDX_INTERRUPTED_RESTARTABLE because SEAMCALL TDH_MEM_PAGE_DEMOTE does not check interrupts (including NMIs) for basic TDX (with or without Dynamic PAMT).

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Yes. But I guess detecting TDX module version or if it supports certain feature is a generic problem. e.g., certain versions of TDX module have bugs in zero-step mitigation and may block vCPU entering.

So, maybe it deserves a separate series?

Quoted 35 lines
Quoted 35 lines

---
RFC v2:
- Refine the patch log (Rick).
- Do not handle TDX_INTERRUPTED_RESTARTABLE as the new TDX modules in planning do not check interrupts for basic TDX.

RFC v1:
- Rebased and split patch. Updated patch log.

---
  arch/x86/include/asm/tdx.h  |  2 ++
  arch/x86/virt/vmx/tdx/tdx.c | 20 ++++++++++++++++++++
  arch/x86/virt/vmx/tdx/tdx.h |  1 +
  3 files changed, 23 insertions(+)
arch/x86/include/asm/tdx.h +2 -0
index f968b736871a..d2cf48e273d5 100644
@@ -178,6 +178,8 @@ u64 tdh_mng_key_config(struct tdx_td *td);
  u64 tdh_mng_create(struct tdx_td *td, u16 hkid);
  u64 tdh_vp_create(struct tdx_td *td, struct tdx_vp *vp);
  u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data);
+u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
+			u64 *ext_err1, u64 *ext_err2);
  u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2);
  u64 tdh_mr_finalize(struct tdx_td *td);
  u64 tdh_vp_flush(struct tdx_vp *vp);
arch/x86/virt/vmx/tdx/tdx.c +0 -1
index 580f14f64822..d941f083f741 100644
--- a/arch/x86/virt/vmx/tdx/tdx.c
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data) · line 1827

Quoted 5 lines

+++ b/arch/x86/virt/vmx/tdx/tdx.c

arch/x86/virt/vmx/tdx/tdx.c +1 -0
@@ -1825,6 +1825,26 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data)
  }
  EXPORT_SYMBOL_GPL(tdh_mng_rd);
+u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,

Nit: Is it better to use a var name that clearly tell that the page is used as a table page?

Yes, Thanks!
I also plan to do it (as well as for that tdx_spte_demote_private_spte() as mentioned in
https://lore.kernel.org/all/[email protected]).

Quoted 36 lines
Quoted 34 lines
arch/x86/virt/vmx/tdx/tdx.c +19 -0
+			u64 *ext_err1, u64 *ext_err2)
+{
+	struct tdx_module_args args = {
+		.rcx = gpa | level,
+		.rdx = tdx_tdr_pa(td),
+		.r8 = page_to_phys(page),
+	};
+	u64 ret;
+
+	tdx_clflush_page(page);
+	ret = seamcall_ret(TDH_MEM_PAGE_DEMOTE, &args);
+
+	*ext_err1 = args.rcx;
+	*ext_err2 = args.rdx;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tdh_mem_page_demote);
+
  u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2)
  {
  	struct tdx_module_args args = {
arch/x86/virt/vmx/tdx/tdx.h +1 -0
index 096c78a1d438..a6c0fa53ece9 100644
@@ -24,6 +24,7 @@
  #define TDH_MNG_KEY_CONFIG		8
  #define TDH_MNG_CREATE			9
  #define TDH_MNG_RD			11
+#define TDH_MEM_PAGE_DEMOTE		15
  #define TDH_MR_EXTEND			16
  #define TDH_MR_FINALIZE			17
  #define TDH_VP_FLUSH			18
#5 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 8 lines · Yan Zhao
Quoted 4 lines

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Yes. But I guess detecting TDX module version or if it supports certain feature is a generic problem. e.g., certain versions of TDX module have bugs in zero-step mitigation and may block vCPU entering.

We had talked in the past of not checking versions because it would require KVM to keep logic of which features in which TDX module.

If there is a flag we could check it, but we did not ask for one here. We already have a situation where there are bug fixes that KVM depends on, with no way to check.

I guess the difference here is that if the behavior is missing, KVM has an option to continue with just small pages. But at the same time, huge pages is very likely to succeed in either case. The "feature" is closer to closing a theoretical race. So very much like the many bugs we don't check for. I'm leaning towards lumping it into that category. And we can add "how do we want to check for TDX module bugs" to the arch todo list. But it's probably down the list, if we even want to do anything.

What do you think?

#6 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 5 lines · Rick P Edgecombe
Quoted 4 lines · Yan Zhao
Quoted 4 lines

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Quoted 7 lines
Quoted 4 lines

Yes. But I guess detecting TDX module version or if it supports certain feature is a generic problem. e.g., certain versions of TDX module have bugs in zero-step mitigation and may block vCPU entering.

We had talked in the past of not checking versions because it would require KVM to keep logic of which features in which TDX module.

Checking for features is different from refusing to load broken modules. I don't want KVM to rely on version numbers to query features, because that relies on "newer" module versions always being a superset relative to "older" versions.

Quoted 12 lines

If there is a flag we could check it, but we did not ask for one here. We already have a situation where there are bug fixes that KVM depends on, with no way to check.

I guess the difference here is that if the behavior is missing, KVM has an option to continue with just small pages. But at the same time, huge pages is very likely to succeed in either case. The "feature" is closer to closing a theoretical race. So very much like the many bugs we don't check for. I'm leaning towards lumping it into that category. And we can add "how do we want to check for TDX module bugs" to the arch todo list. But it's probably down the list, if we even want to do anything.

Quoted 1 line

What do you think?

Could we taint the kernel and print a scary message if a known-buggy TDX module is loaded?

#7 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 12 lines · Sean Christopherson
Quoted 12 lines

If there is a flag we could check it, but we did not ask for one here. We already have a situation where there are bug fixes that KVM depends on, with no way to check.

I guess the difference here is that if the behavior is missing, KVM has an option to continue with just small pages. But at the same time, huge pages is very likely to succeed in either case. The "feature" is closer to closing a theoretical race. So very much like the many bugs we don't check for. I'm leaning towards lumping it into that category. And we can add "how do we want to check for TDX module bugs" to the arch todo list. But it's probably down the list, if we even want to do anything.

Quoted 4 lines
Quoted 1 line

What do you think?

Could we taint the kernel and print a scary message if a known-buggy TDX module is loaded?

If we know which TDX modules have bugs, I guess. There may be some bugs that only affect the guest, where tainting would not be appropriate. Probably would want to do it at TDX module load time, so that people that don't use TDX don't get their kernel tainted from an old TDX module in the BIOS.

What would you want a TDX module interface for this to look like? Like a bitmap of fixed bugs? KVM keeps a list of bugs it cares about and compares it to the list provided by TDX module? I think it could work if KVM is ok selecting and keeping a bitmap of TDX module bugs.

author
#8 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 23 lines · Rick P
Quoted 12 lines · Sean Christopherson
Quoted 12 lines

If there is a flag we could check it, but we did not ask for one here. We already have a situation where there are bug fixes that KVM depends on, with no way to check.

I guess the difference here is that if the behavior is missing, KVM has an option to continue with just small pages. But at the same time, huge pages is very likely to succeed in either case. The "feature" is closer to closing a theoretical race. So very much like the many bugs we don't check for. I'm leaning towards lumping it into that category. And we can add "how do we want to check for TDX module bugs" to the arch todo list. But it's probably down the list, if we even want to do anything.

Quoted 4 lines
Quoted 1 line

What do you think?

Could we taint the kernel and print a scary message if a known-buggy TDX module is loaded?

If we know which TDX modules have bugs, I guess. There may be some bugs that only affect the guest, where tainting would not be appropriate. Probably would want to do it at TDX module load time, so that people that don't use TDX don't get their kernel tainted from an old TDX module in the BIOS.

Quoted 4 lines

What would you want a TDX module interface for this to look like? Like a bitmap of fixed bugs? KVM keeps a list of bugs it cares about and compares it to the list provided by TDX module? I think it could work if KVM is ok selecting and keeping a bitmap of TDX module bugs.

Specific to the problem of TDX_INTERRUPTED_RESTARTABLE, could we choose to port this feature to all old TDX modules?

#9 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

In reply to Yan Zhao
Quoted 8 lines · Yan Zhao
Quoted 8 lines
Quoted 3 lines

Do not handle TDX_INTERRUPTED_RESTARTABLE because SEAMCALL TDH_MEM_PAGE_DEMOTE does not check interrupts (including NMIs) for basic TDX (with or without Dynamic PAMT).

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Quoted 5 lines

Yes. But I guess detecting TDX module version or if it supports certain feature is a generic problem. e.g., certain versions of TDX module have bugs in zero-step mitigation and may block vCPU entering.

So, maybe it deserves a separate series?

Looking at the spec (TDX module ABI spec 348551-007US), is it enumerated via TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY?

5.4.25.3.9.

Interruptibility

If the TD is not partitioned (i.e., it has been configured with no L2 VMs), and the TDX Module enumerates TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY as 1, TDH.MEM.PAGE.DEMOTE is not interruptible.

So if the decision is to not use 2M page when TDH_MEM_PAGE_DEMOTE can return TDX_INTERRUPTED_RESTARTABLE, maybe we can just check this enumeration in fault handler and always make mapping level as 4K?

author
#10 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 9 lines · Kai
Quoted 8 lines · Yan Zhao
Quoted 8 lines
Quoted 3 lines

Do not handle TDX_INTERRUPTED_RESTARTABLE because SEAMCALL TDH_MEM_PAGE_DEMOTE does not check interrupts (including NMIs) for basic TDX (with or without Dynamic PAMT).

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Quoted 8 lines
Quoted 5 lines

Yes. But I guess detecting TDX module version or if it supports certain feature is a generic problem. e.g., certain versions of TDX module have bugs in zero-step mitigation and may block vCPU entering.

So, maybe it deserves a separate series?

Looking at the spec (TDX module ABI spec 348551-007US), is it enumerated via TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY?

Yes. I checked the unreleased TDX module code that enumerates this bit (starting from version TDX_1.5.28.00.972). TDH.MEM.PAGE.DEMOTE will not return TDX_INTERRUPTED_RESTARTABLE for L1 VMs.

Quoted 4 lines

5.4.25.3.9.

Interruptibility

Quoted 8 lines

If the TD is not partitioned (i.e., it has been configured with no L2 VMs), and the TDX Module enumerates TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY as 1, TDH.MEM.PAGE.DEMOTE is not interruptible.

So if the decision is to not use 2M page when TDH_MEM_PAGE_DEMOTE can return TDX_INTERRUPTED_RESTARTABLE, maybe we can just check this enumeration in fault handler and always make mapping level as 4K?

Thanks for this info! I think this is a very good idea and the right direction. If no objection, I'll update the code in this way.

#11 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 13 lines · Yan Zhao
Quoted 12 lines · Kai
Quoted 11 lines · Yan Zhao
Quoted 7 lines
Quoted 3 lines

Do not handle TDX_INTERRUPTED_RESTARTABLE because SEAMCALL TDH_MEM_PAGE_DEMOTE does not check interrupts (including NMIs) for basic TDX (with or without Dynamic PAMT).

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Yes. But I guess detecting TDX module version or if it supports certain feature is a generic problem. e.g., certain versions of TDX module have bugs in zero-step mitigation and may block vCPU entering.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

So, maybe it deserves a separate series?

Looking at the spec (TDX module ABI spec 348551-007US), is it enumerated via TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY?

Yes. I checked the unreleased TDX module code that enumerates this bit (starting from version TDX_1.5.28.00.972). TDH.MEM.PAGE.DEMOTE will not return TDX_INTERRUPTED_RESTARTABLE for L1 VMs.

According to the content pasted by Kai below, it just says there will be no TDX_INTERRUPTED_RESTARTABLE for TDH.MEM.PAGE.DEMOTE if no L2 VMs.

KVM doesn't support TD partition yet, just for clarification,  what if the demotion is for L1 VM, but there are L2 VMs configured?

Quoted 18 lines
Quoted 5 lines

5.4.25.3.9.

Interruptibility

Quoted 8 lines

If the TD is not partitioned (i.e., it has been configured with no L2 VMs), and the TDX Module enumerates TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY as 1, TDH.MEM.PAGE.DEMOTE is not interruptible.

So if the decision is to not use 2M page when TDH_MEM_PAGE_DEMOTE can return TDX_INTERRUPTED_RESTARTABLE, maybe we can just check this enumeration in fault handler and always make mapping level as 4K?

Thanks for this info! I think this is a very good idea and the right direction. If no objection, I'll update the code in this way.

author
#12 lore raw

Re: [RFC PATCH v2 02/23] x86/virt/tdx: Add SEAMCALL wrapper tdh_mem_page_demote()

Quoted 23 lines · Binbin Wu
Quoted 13 lines · Yan Zhao
Quoted 12 lines · Kai
Quoted 11 lines · Yan Zhao
Quoted 7 lines
Quoted 3 lines

Do not handle TDX_INTERRUPTED_RESTARTABLE because SEAMCALL TDH_MEM_PAGE_DEMOTE does not check interrupts (including NMIs) for basic TDX (with or without Dynamic PAMT).

The cover letter mentions that there is a new TDX module in planning, which disables the interrupt checking. I guess TDX module would need to have a interface to report the change, KVM then decides to enable huge page support or not for TDs?

Yes. But I guess detecting TDX module version or if it supports certain feature is a generic problem. e.g., certain versions of TDX module have bugs in zero-step mitigation and may block vCPU entering.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

So, maybe it deserves a separate series?

Looking at the spec (TDX module ABI spec 348551-007US), is it enumerated via TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY?

Yes. I checked the unreleased TDX module code that enumerates this bit (starting from version TDX_1.5.28.00.972). TDH.MEM.PAGE.DEMOTE will not return TDX_INTERRUPTED_RESTARTABLE for L1 VMs.

Quoted 5 lines

According to the content pasted by Kai below, it just says there will be no TDX_INTERRUPTED_RESTARTABLE for TDH.MEM.PAGE.DEMOTE if no L2 VMs.

KVM doesn't support TD partition yet, just for clarification,  what if the demotion is for L1 VM, but there are L2 VMs configured?

Right. The description pasted by Kai is more accurate:
"There will be no TDX_INTERRUPTED_RESTARTABLE for TDH.MEM.PAGE.DEMOTE if no L2
VMs".

From the code, DEMOTE may return TDX_INTERRUPTED_RESTARTABLE if tdcs_ptr->management_fields.num_l2_vms is non-zero.

Thanks for flagging this.

Quoted 18 lines
Quoted 17 lines
Quoted 4 lines

5.4.25.3.9.

Interruptibility

Quoted 8 lines

If the TD is not partitioned (i.e., it has been configured with no L2 VMs), and the TDX Module enumerates TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY as 1, TDH.MEM.PAGE.DEMOTE is not interruptible.

So if the decision is to not use 2M page when TDH_MEM_PAGE_DEMOTE can return TDX_INTERRUPTED_RESTARTABLE, maybe we can just check this enumeration in fault handler and always make mapping level as 4K?

Thanks for this info! I think this is a very good idea and the right direction. If no objection, I'll update the code in this way.

author
#13 lore raw

RFC PATCH v2 3/23 x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

In reply to Yan Zhao

After removing a TD's private page, the TDX module does not write back and invalidate cache lines associated with the page and its keyID (i.e., the TD's guest keyID). The SEAMCALL wrapper tdh_phymem_page_wbinvd_hkid() enables the caller to provide the TD's guest keyID and physical memory address to invoke the SEAMCALL TDH_PHYMEM_PAGE_WBINVD to perform cache line invalidation.

Enhance the SEAMCALL wrapper tdh_phymem_page_wbinvd_hkid() to support cache line invalidation for huge pages by introducing the parameters "folio", "start_idx", and "npages". These parameters specify the physical memory starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs. Return TDX_OPERAND_INVALID if the specified memory is not entirely contained within a single folio.

---
RFC v2:
- Enhance tdh_phymem_page_wbinvd_hkid() to invalidate multiple pages directly, rather than looping within KVM, following Dave's suggestion:
"Don't wrap the wrappers." (Rick).

RFC v1:
- Split patch
- Aded a helper tdx_wbinvd_page() in TDX, which accepts param "struct page *".

---
 arch/x86/include/asm/tdx.h  |  4 ++--
 arch/x86/kvm/vmx/tdx.c      |  6 ++++--
 arch/x86/virt/vmx/tdx/tdx.c | 17 ++++++++++++++---
 3 files changed, 20 insertions(+), 7 deletions(-)
arch/x86/include/asm/tdx.h +2 -2
index d2cf48e273d5..a125bb20a28a 100644
@@ -194,8 +194,8 @@ u64 tdh_mem_track(struct tdx_td *tdr);
 u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, u64 level, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_phymem_cache_wb(bool resume);
 u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td);
-u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page);
-
+u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
+				unsigned long start_idx, unsigned long npages);
 void tdx_meminfo(struct seq_file *m);
 #else
 static inline void tdx_init(void) { }
arch/x86/kvm/vmx/tdx.c +4 -2
index 0a2b183899d8..8eaf8431c5f1 100644
@@ -1694,6 +1694,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 {
 	int tdx_level = pg_level_to_tdx_sept_level(level);
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	struct folio *folio = page_folio(page);
 	gpa_t gpa = gfn_to_gpa(gfn);
 	u64 err, entry, level_state;
 
@@ -1728,8 +1729,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 		return -EIO;
 	}
 
-	err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
-
+	err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, folio,
+					  folio_page_idx(folio, page),
+					  KVM_PAGES_PER_HPAGE(level));
 	if (KVM_BUG_ON(err, kvm)) {
 		pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err);
 		return -EIO;
arch/x86/virt/vmx/tdx/tdx.c +14 -3
index d941f083f741..64219c659844 100644
@@ -2030,13 +2030,24 @@ u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td)
 }
 EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_tdr);
 
-u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
+u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
+				unsigned long start_idx, unsigned long npages)
 {
+	struct page *start = folio_page(folio, start_idx);
 	struct tdx_module_args args = {};
+	u64 err;
+
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;
 
-	args.rcx = mk_keyed_paddr(hkid, page);
+	for (unsigned long i = 0; i < npages; i++) {
+		args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
 
-	return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
+		err = seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
+		if (err)
+			break;
+	}
+	return err;
 }
 EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_hkid);
 
Signature · 1 line
2.43.2
#14 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 14 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +2 -1
-u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
+u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
+				unsigned long start_idx, unsigned long npages)

 {

arch/x86/virt/vmx/tdx/tdx.c +1 -0
+	struct page *start = folio_page(folio, start_idx);

  struct tdx_module_args args = {};

arch/x86/virt/vmx/tdx/tdx.c +4 -0
+	u64 err;
+
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;

arch/x86/virt/vmx/tdx/tdx.c +2 -1
-	args.rcx = mk_keyed_paddr(hkid, page);
+	for (unsigned long i = 0; i < npages; i++) {
+		args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
Quoted 1 line

Just FYI: seems there's a series to remove nth_page() completely:

https://lore.kernel.org/kvm/[email protected]/

author
#15 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 19 lines · Kai
Quoted 14 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +2 -1
-u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
+u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
+				unsigned long start_idx, unsigned long npages)

 {

arch/x86/virt/vmx/tdx/tdx.c +1 -0
+	struct page *start = folio_page(folio, start_idx);

  struct tdx_module_args args = {};

arch/x86/virt/vmx/tdx/tdx.c +4 -0
+	u64 err;
+
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;

arch/x86/virt/vmx/tdx/tdx.c +2 -1
-	args.rcx = mk_keyed_paddr(hkid, page);
+	for (unsigned long i = 0; i < npages; i++) {
+		args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
Quoted 1 line

Just FYI: seems there's a series to remove nth_page() completely:

Quoted 1 line

https://lore.kernel.org/kvm/[email protected]/

Ah, thanks!
Then we can get rid of the "unsigned long i".

- for (unsigned long i = 0; i < npages; i++) {
- args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
+ while (npages--) {
+ args.rcx = mk_keyed_paddr(hkid, start++);

#16 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 25 lines · Yan Zhao
Quoted 19 lines · Kai
Quoted 14 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +2 -1
-u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
+u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
+				unsigned long start_idx, unsigned long npages)

 {

arch/x86/virt/vmx/tdx/tdx.c +1 -0
+	struct page *start = folio_page(folio, start_idx);

  struct tdx_module_args args = {};

arch/x86/virt/vmx/tdx/tdx.c +4 -0
+	u64 err;
+
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;

arch/x86/virt/vmx/tdx/tdx.c +2 -1
-	args.rcx = mk_keyed_paddr(hkid, page);
+	for (unsigned long i = 0; i < npages; i++) {
+		args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
Quoted 1 line

Just FYI: seems there's a series to remove nth_page() completely:

Quoted 1 line

https://lore.kernel.org/kvm/[email protected]/

Ah, thanks!
Then we can get rid of the "unsigned long i".

Review

Quoted 4 lines
(quoted diff) +2 -2
-       for (unsigned long i = 0; i < npages; i++) {
-               args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
+       while (npages--) {
+               args.rcx = mk_keyed_paddr(hkid, start++);

You may want to be careful about doing '++' on a 'struct page *'. I am not expert, but I saw below discussion on the thread [*] which led to the series to get rid of nth_page():

Quoted 5 lines

I wish we didn't have nth_page() at all. I really don't think it's a valid operation. It's been around forever, but I think it was broken as introduced, exactly because I don't think you can validly even have allocations that cross section boundaries.

Ordinary buddy allocations cannot exceed a memory section, but hugetlb and dax can with gigantic folios ... :(

We had some weird bugs with that, because people keep forgetting that you cannot just use page++ unconditionally with such folios.

So, why not just get the actual page for each index within the loop?

[*]:
https://lore.kernel.org/all/CAHk-=wiCYfNp4AJLBORU-c7ZyRBUp66W2-Et6cdQ4REx-GyQ_A@mail.gmail.com/T/#m49ba78f5f630b27fa6d3d0737271f047af599c60

author
#17 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 25 lines · Kai
Quoted 24 lines · Yan Zhao
Quoted 19 lines · Kai
Quoted 14 lines · Yan Zhao
arch/x86/virt/vmx/tdx/tdx.c +2 -1
-u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct page *page)
+u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
+				unsigned long start_idx, unsigned long npages)

 {

arch/x86/virt/vmx/tdx/tdx.c +1 -0
+	struct page *start = folio_page(folio, start_idx);

  struct tdx_module_args args = {};

arch/x86/virt/vmx/tdx/tdx.c +4 -0
+	u64 err;
+
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;

arch/x86/virt/vmx/tdx/tdx.c +2 -1
-	args.rcx = mk_keyed_paddr(hkid, page);
+	for (unsigned long i = 0; i < npages; i++) {
+		args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
Quoted 1 line

Just FYI: seems there's a series to remove nth_page() completely:

Quoted 1 line

https://lore.kernel.org/kvm/[email protected]/

Ah, thanks!
Then we can get rid of the "unsigned long i".

Quoted 7 lines

Review

Quoted 5 lines
(quoted diff) +2 -2
-       for (unsigned long i = 0; i < npages; i++) {
-               args.rcx = mk_keyed_paddr(hkid, nth_page(start, i));
+       while (npages--) {
+               args.rcx = mk_keyed_paddr(hkid, start++);

You may want to be careful about doing '++' on a 'struct page *'. I am not

Before the removing nth_page() series, linux kernel defines nth_page() like this:

  #if defined(CONFIG_SPARSEMEM) && !defined(CONFIG_SPARSEMEM_VMEMMAP)
  #define nth_page(page,n) pfn_to_page(page_to_pfn((page)) + (n))
  #define folio_page_idx(folio, p)        (page_to_pfn(p) - folio_pfn(folio))
  #else
  #define nth_page(page,n) ((page) + (n))
  #define folio_page_idx(folio, p)        ((p) - &(folio)->page)
  #endif

i.e., unless SPARSEMEM without SPARSEMEM_VMEMMAP, a folio's page is contiguous.

In David's removing nth_page() series, CONFIG_SPARSEMEM_VMEMMAP is auto-selected along with CONFIG_SPARSEMEM in all architectures but sh.

David further ensures folio pages are continuous even on sh with the problematic kernel configs (i.e., SPARSEMEM without SPARSEMEM_VMEMMAP) [1]:

: Currently, only a single architectures supports ARCH_HAS_GIGANTIC_PAGE : but not SPARSEMEM_VMEMMAP: sh.
:
: Fortunately, the biggest hugetlb size sh supports is 64 MiB
: (HUGETLB_PAGE_SIZE_64MB) and the section size is at least 64 MiB : (SECTION_SIZE_BITS == 26), so their use case is not degraded. :
: As folios and memory sections are naturally aligned to their order-2 size : in memory, consequently a single folio can no longer span multiple memory : sections on these problematic kernel configs.

So it's safe to assume folio pages are continuous.

[1] https://lore.kernel.org/kvm/[email protected]/

Quoted 7 lines

expert, but I saw below discussion on the thread [*] which led to the series to get rid of nth_page():

Quoted 4 lines

I wish we didn't have nth_page() at all. I really don't think it's a valid operation. It's been around forever, but I think it was broken as introduced, exactly because I don't think you can validly even have allocations that cross section boundaries.

Quoted 5 lines

Ordinary buddy allocations cannot exceed a memory section, but hugetlb and dax can with gigantic folios ... :(

We had some weird bugs with that, because people keep forgetting that you cannot just use page++ unconditionally with such folios.

I found Linus's reply to David [2] :
: On Tue, 5 Aug 2025 at 16:37, David Hildenbrand <[email protected]> wrote:
: >
: > Ordinary buddy allocations cannot exceed a memory section, but hugetlb and : > dax can with gigantic folios ... :(
:
: Just turn that code off. Nobody sane cares.
:
: It sounds like people have bent over backwards to fix the insane case : instead of saying "that's insane, let's not support it".
:
: And yes, "that's insane" is actually fairly recent. It's not that long : ago that we made SPARSEMEM_VMEMMAP the mandatory option on x86-64. So : it was all sane in a historical context, but it's not sane any more. :
: But now it *is* the mandatory option both on x86 and arm64, so I : really think it's time to get rid of pointless pain points.
:
: (I think powerpc still makes it an option to do sparsemem without : vmemmap, but it *is* an option there too)

The removing nth_page() series then ensures hugetlb and dax are Ok like changes in [3]. The series then iterates over all pages in a hugetlb folio by invoking page++. e.g., [4][5].

[2] https://lore.kernel.org/all/CAHk-=wiYLcax-5THGofwk-SAWYZ1RsP08b+rozXOm0wZRCE9UQ@mail.gmail.com
[3] https://lore.kernel.org/kvm/[email protected]
[4] https://lore.kernel.org/kvm/[email protected]
[5] https://lore.kernel.org/kvm/[email protected]

Quoted 1 line

So, why not just get the actual page for each index within the loop?

We need to invoke folio_page() to get the actual page.

In [6], the new folio_page() implementation is

static inline struct page *folio_page(struct folio *folio, unsigned long n)

{
	return &folio->page + n;
}

So, invoking folio_page() should be equal to page++ in our case.

[6] https://lore.kernel.org/kvm/[email protected]

Quoted 2 lines

[*]:
https://lore.kernel.org/all/CAHk-=wiCYfNp4AJLBORU-c7ZyRBUp66W2-Et6cdQ4REx-GyQ_A@mail.gmail.com/T/#m49ba78f5f630b27fa6d3d0737271f047af599c60

#18 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Thanks for all the explanation.

[...]

Quoted 10 lines

In [6], the new folio_page() implementation is

static inline struct page *folio_page(struct folio *folio, unsigned long n)

{
	return &folio->page + n;
}

So, invoking folio_page() should be equal to page++ in our case.

[6] https://lore.kernel.org/kvm/[email protected]

Sure. But it seems you will need to wait all patches that you mentioned to be merged to safely use 'page++' for pages in a folio?

And if you do:

	for (i = 0; i < npages; i++)
	{
		struct page *p = folio_page(folio, start_idx + i);
		struct tdx_module_args args = {};

		args.rcx = mk_keyed_paddr(hkid, p);
		...
	}

It should work w/o any dependency?

Anyway, I don't have any strong opinion, as long as it works. You may choose what you want. :-)

author
#19 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 13 lines · Kai

Thanks for all the explanation.

[...]

Quoted 7 lines

In [6], the new folio_page() implementation is

static inline struct page *folio_page(struct folio *folio, unsigned long n)

{
	return &folio->page + n;
}
Quoted 6 lines
Quoted 3 lines

So, invoking folio_page() should be equal to page++ in our case.

[6] https://lore.kernel.org/kvm/[email protected]

Sure. But it seems you will need to wait all patches that you mentioned to be merged to safely use 'page++' for pages in a folio?

Correct.

Quoted 11 lines

And if you do:

	for (i = 0; i < npages; i++)
	{
		struct page *p = folio_page(folio, start_idx + i);
		struct tdx_module_args args = {};

		args.rcx = mk_keyed_paddr(hkid, p);
		...
	}
Quoted 4 lines

It should work w/o any dependency?

Anyway, I don't have any strong opinion, as long as it works. You may choose what you want. :-)

I don't have a strong opinion either. However, based on previous reviews, it seems people prefer while (npages--) over introducing an additional variable. I guess I'll choose a version depending on whether these patches are merged when I post the next version. :)

#20 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

In reply to Huang, Kai
Quoted 14 lines · Kai

Sure. But it seems you will need to wait all patches that you mentioned to be merged to safely use 'page++' for pages in a folio?

And if you do:

	for (i = 0; i < npages; i++)
	{
		struct page *p = folio_page(folio, start_idx + i);
		struct tdx_module_args args = {};

		args.rcx = mk_keyed_paddr(hkid, p);
		...
	}
Quoted 4 lines

It should work w/o any dependency?

Anyway, I don't have any strong opinion, as long as it works. You may choose what you want. 🙂

Folks, I'll make it easy: Do what Kai suggested above. It works universally and it's obvious. Saving an "i" variable only makes the code harder to read.

If anyone thinks that:

while (npages--)

Is easier to understand than the most common C idiom on the planet:

	for (i = 0; i < npages; i++)

... then I don't know what to tell them.

author
#21 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 26 lines · Dave Hansen
Quoted 14 lines · Kai

Sure. But it seems you will need to wait all patches that you mentioned to be merged to safely use 'page++' for pages in a folio?

And if you do:

	for (i = 0; i < npages; i++)
	{
		struct page *p = folio_page(folio, start_idx + i);
		struct tdx_module_args args = {};

		args.rcx = mk_keyed_paddr(hkid, p);
		...
	}
Quoted 4 lines

It should work w/o any dependency?

Anyway, I don't have any strong opinion, as long as it works. You may choose what you want. 🙂

Folks, I'll make it easy: Do what Kai suggested above. It works universally and it's obvious. Saving an "i" variable only makes the code harder to read.

If anyone thinks that:

Quoted 7 lines

while (npages--)

Is easier to understand than the most common C idiom on the planet:

	for (i = 0; i < npages; i++)

... then I don't know what to tell them.

Got it. That's very helpful. Thanks for the instruction!

#22 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

In reply to Yan Zhao
Quoted 13 lines · Yan Zhao

index 0a2b183899d8..8eaf8431c5f1 100644

arch/x86/kvm/vmx/tdx.c +1 -0
@@ -1694,6 +1694,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       struct folio *folio = page_folio(page);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
 

arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1734

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +3 -2
@@ -1728,8 +1729,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
                return -EIO;
        }
 
-       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
-
+       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, folio,
+                                         folio_page_idx(folio, page),
+                                         KVM_PAGES_PER_HPAGE(level));

This code seems to assume that folio_order() always matches the level at which it is mapped in the EPT entries. IIUC guest_memfd can decide to split folios to 4K for the complete huge folio before zapping the hugepage EPT mappings. I think it's better to just round the pfn to the hugepage address based on the level they were mapped at instead of relying on the folio order.

author
#23 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 17 lines · Vishal Annapurve
Quoted 16 lines · Yan Zhao

index 0a2b183899d8..8eaf8431c5f1 100644

arch/x86/kvm/vmx/tdx.c +1 -0
@@ -1694,6 +1694,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       struct folio *folio = page_folio(page);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
 
@@ -1728,8 +1729,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
                return -EIO;
        }
 
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1734

Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +3 -2
-       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
-
+       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, folio,
+                                         folio_page_idx(folio, page),
+                                         KVM_PAGES_PER_HPAGE(level));

This code seems to assume that folio_order() always matches the level at which it is mapped in the EPT entries.

I don't think so.
Please check the implemenation of tdh_phymem_page_wbinvd_hkid() [1]. Only npages=KVM_PAGES_PER_HPAGE(level) will be invalidated, while npages <= folio_nr_pages(folio).

[1] https://lore.kernel.org/all/[email protected]/

Quoted 5 lines

IIUC guest_memfd can decide
to split folios to 4K for the complete huge folio before zapping the hugepage EPT mappings. I think it's better to just round the pfn to the hugepage address based on the level they were mapped at instead of relying on the folio order.

#24 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 27 lines · Yan Zhao
Quoted 17 lines · Vishal Annapurve
Quoted 16 lines · Yan Zhao

index 0a2b183899d8..8eaf8431c5f1 100644

arch/x86/kvm/vmx/tdx.c +1 -0
@@ -1694,6 +1694,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       struct folio *folio = page_folio(page);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
 
@@ -1728,8 +1729,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
                return -EIO;
        }
 
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1734

Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +3 -2
-       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
-
+       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, folio,
+                                         folio_page_idx(folio, page),
+                                         KVM_PAGES_PER_HPAGE(level));

This code seems to assume that folio_order() always matches the level at which it is mapped in the EPT entries.

Quoted 4 lines

I don't think so.
Please check the implemenation of tdh_phymem_page_wbinvd_hkid() [1]. Only npages=KVM_PAGES_PER_HPAGE(level) will be invalidated, while npages <= folio_nr_pages(folio).

Is the gfn passed to tdx_sept_drop_private_spte() always huge page aligned if mapping is at huge page granularity?

If gfn/pfn is not aligned then when folio is split to 4K, page_folio() will return the same page and folio_order and folio_page_idx() will be zero. This can cause tdh_phymem_page_wbinvd_hkid() to return failure.

If the expectation is that page_folio() will always point to a head page for given hugepage granularity mapping then that logic will not work correctly IMO.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/

Quoted 5 lines

IIUC guest_memfd can decide
to split folios to 4K for the complete huge folio before zapping the hugepage EPT mappings. I think it's better to just round the pfn to the hugepage address based on the level they were mapped at instead of relying on the folio order.

author
#25 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 28 lines · Vishal Annapurve
Quoted 27 lines · Yan Zhao
Quoted 25 lines · Vishal Annapurve
Quoted 13 lines · Yan Zhao

index 0a2b183899d8..8eaf8431c5f1 100644

arch/x86/kvm/vmx/tdx.c +1 -0
@@ -1694,6 +1694,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       struct folio *folio = page_folio(page);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
 

arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1734

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +3 -2
@@ -1728,8 +1729,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
                return -EIO;
        }
 
-       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
-
+       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, folio,
+                                         folio_page_idx(folio, page),
+                                         KVM_PAGES_PER_HPAGE(level));

This code seems to assume that folio_order() always matches the level at which it is mapped in the EPT entries.

Quoted 7 lines
Quoted 4 lines

I don't think so.
Please check the implemenation of tdh_phymem_page_wbinvd_hkid() [1]. Only npages=KVM_PAGES_PER_HPAGE(level) will be invalidated, while npages <= folio_nr_pages(folio).

Is the gfn passed to tdx_sept_drop_private_spte() always huge page aligned if mapping is at huge page granularity?

Yes.
The GFN passed to tdx_sept_set_private_spte() is huge page aligned in kvm_tdp_mmu_map(). SEAMCALL TDH_MEM_PAGE_AUG will also fail otherwise. The GFN passed to tdx_sept_remove_private_spte() comes from the same mapping entry in the mirror EPT.

Quoted 7 lines

If gfn/pfn is not aligned then when folio is split to 4K, page_folio() will return the same page and folio_order and folio_page_idx() will be zero. This can cause tdh_phymem_page_wbinvd_hkid() to return failure.

If the expectation is that page_folio() will always point to a head page for given hugepage granularity mapping then that logic will not work correctly IMO.

The current logic is that:
1. tdh_mem_page_aug() maps physical memory starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.
(npages corresponds to the mapping level KVM_PAGES_PER_HPAGE(level)). e.g. it can map at level 2MB, starting from the 4MB offset in a folio of order 1GB.

2. if split occurs, the huge 2MB mapping will be split into 4KB ones, while the underlying folio remains 1GB.
e.g. now the 0th 4KB mapping after split points to the 4MB offset in the 1GB folio, and the 1st 4KB mapping points to the 4MB+4KB offset... The mapping level after split is 4KB.

3. tdx_sept_remove_private_spte() invokes tdh_mem_page_remove() and tdh_phymem_page_wbinvd_hkid().
-The GFN is 2MB aligned and level is 2MB if split does not occur or -The GFN is 4KB aligned and level is 4KB if split has occurred. While the underlying folio remains 1GB, the folio_page_idx(folio, page) specifies the offset in the folio, and the npages corresponding to the mapping level is <= folio_nr_pages(folio).

Quoted 7 lines
Quoted 7 lines

[1] https://lore.kernel.org/all/[email protected]/

Quoted 5 lines

IIUC guest_memfd can decide
to split folios to 4K for the complete huge folio before zapping the hugepage EPT mappings. I think it's better to just round the pfn to the hugepage address based on the level they were mapped at instead of relying on the folio order.

#26 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 57 lines · Yan Zhao
Quoted 28 lines · Vishal Annapurve
Quoted 27 lines · Yan Zhao
Quoted 25 lines · Vishal Annapurve
Quoted 13 lines · Yan Zhao

index 0a2b183899d8..8eaf8431c5f1 100644

arch/x86/kvm/vmx/tdx.c +1 -0
@@ -1694,6 +1694,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       struct folio *folio = page_folio(page);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
 

arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1734

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +3 -2
@@ -1728,8 +1729,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
                return -EIO;
        }
 
-       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
-
+       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, folio,
+                                         folio_page_idx(folio, page),
+                                         KVM_PAGES_PER_HPAGE(level));

This code seems to assume that folio_order() always matches the level at which it is mapped in the EPT entries.

Quoted 7 lines
Quoted 4 lines

I don't think so.
Please check the implemenation of tdh_phymem_page_wbinvd_hkid() [1]. Only npages=KVM_PAGES_PER_HPAGE(level) will be invalidated, while npages <= folio_nr_pages(folio).

Is the gfn passed to tdx_sept_drop_private_spte() always huge page aligned if mapping is at huge page granularity?

Yes.
The GFN passed to tdx_sept_set_private_spte() is huge page aligned in kvm_tdp_mmu_map(). SEAMCALL TDH_MEM_PAGE_AUG will also fail otherwise. The GFN passed to tdx_sept_remove_private_spte() comes from the same mapping entry in the mirror EPT.

Quoted 7 lines

If gfn/pfn is not aligned then when folio is split to 4K, page_folio() will return the same page and folio_order and folio_page_idx() will be zero. This can cause tdh_phymem_page_wbinvd_hkid() to return failure.

If the expectation is that page_folio() will always point to a head page for given hugepage granularity mapping then that logic will not work correctly IMO.

The current logic is that:
1. tdh_mem_page_aug() maps physical memory starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.
(npages corresponds to the mapping level KVM_PAGES_PER_HPAGE(level)). e.g. it can map at level 2MB, starting from the 4MB offset in a folio of order 1GB.

Quoted 2 lines

2. if split occurs, the huge 2MB mapping will be split into 4KB ones, while the underlying folio remains 1GB.

Private to shared conversion flow discussed so far [1][2][3]:
1) Preallocate maple tree entries needed for conversion
2) Split filemap range being converted to 4K pages
3) Mark KVM MMU invalidation begin for the huge page aligned range
4) Zap KVM MMU entries for the converted range
5) Update maple tree entries to carry final attributes
6) Mark KVM MMU invalidation end for huge page aligned range

Possible addition of splitting cross boundary leafs with the above flow:
1) Preallocate maple tree entries needed for conversion
2) Split filemap range being converted to 4K pages
3) Mark KVM MMU invalidation begin for the huge page aligned range
4) Split KVM MMU private boundary leafs for converted range
5) Zap KVM MMU entries for the converted range
6) Update maple tree entries to carry final attributes
7) Mark KVM MMU invalidation end for huge page aligned range

Note that in both the above flows KVM MMU entries will get zapped after folio is split to 4K i.e. when tdx_sept_remove_private_spte() happens folio will be split but the EPT entry level will still be 2M and the assumption of EPT entries always being subset of folios will not hold true.

I think things might be simplified if KVM TDX stack always operates on the pages without assuming ranges being covered by "folios".

[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://lore.kernel.org/kvm/[email protected]/
[3] https://github.com/googleprodkernel/linux-cc/blob/9ee2bd65cc9b63c871f8f49d217a7a70576a942d/virt/kvm/guest_memfd.c#L894

Quoted 20 lines

e.g. now the 0th 4KB mapping after split points to the 4MB offset in the 1GB folio, and the 1st 4KB mapping points to the 4MB+4KB offset... The mapping level after split is 4KB.

3. tdx_sept_remove_private_spte() invokes tdh_mem_page_remove() and tdh_phymem_page_wbinvd_hkid().
-The GFN is 2MB aligned and level is 2MB if split does not occur or -The GFN is 4KB aligned and level is 4KB if split has occurred. While the underlying folio remains 1GB, the folio_page_idx(folio, page) specifies the offset in the folio, and the npages corresponding to the mapping level is <= folio_nr_pages(folio).

Quoted 7 lines
Quoted 7 lines

[1] https://lore.kernel.org/all/[email protected]/

Quoted 5 lines

IIUC guest_memfd can decide
to split folios to 4K for the complete huge folio before zapping the hugepage EPT mappings. I think it's better to just round the pfn to the hugepage address based on the level they were mapped at instead of relying on the folio order.

author
#27 lore raw

Re: [RFC PATCH v2 03/23] x86/tdx: Enhance tdh_phymem_page_wbinvd_hkid() to invalidate huge pages

Quoted 78 lines · Vishal Annapurve
Quoted 57 lines · Yan Zhao
Quoted 28 lines · Vishal Annapurve
Quoted 27 lines · Yan Zhao
Quoted 25 lines · Vishal Annapurve
Quoted 13 lines · Yan Zhao

index 0a2b183899d8..8eaf8431c5f1 100644

arch/x86/kvm/vmx/tdx.c +1 -0
@@ -1694,6 +1694,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       struct folio *folio = page_folio(page);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
 

arch/x86/kvm/vmx/tdx.c static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn, · line 1734

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +3 -2
@@ -1728,8 +1729,9 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
                return -EIO;
        }
 
-       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, page);
-
+       err = tdh_phymem_page_wbinvd_hkid((u16)kvm_tdx->hkid, folio,
+                                         folio_page_idx(folio, page),
+                                         KVM_PAGES_PER_HPAGE(level));

This code seems to assume that folio_order() always matches the level at which it is mapped in the EPT entries.

Quoted 7 lines
Quoted 4 lines

I don't think so.
Please check the implemenation of tdh_phymem_page_wbinvd_hkid() [1]. Only npages=KVM_PAGES_PER_HPAGE(level) will be invalidated, while npages <= folio_nr_pages(folio).

Is the gfn passed to tdx_sept_drop_private_spte() always huge page aligned if mapping is at huge page granularity?

Yes.
The GFN passed to tdx_sept_set_private_spte() is huge page aligned in kvm_tdp_mmu_map(). SEAMCALL TDH_MEM_PAGE_AUG will also fail otherwise. The GFN passed to tdx_sept_remove_private_spte() comes from the same mapping entry in the mirror EPT.

Quoted 7 lines

If gfn/pfn is not aligned then when folio is split to 4K, page_folio() will return the same page and folio_order and folio_page_idx() will be zero. This can cause tdh_phymem_page_wbinvd_hkid() to return failure.

If the expectation is that page_folio() will always point to a head page for given hugepage granularity mapping then that logic will not work correctly IMO.

The current logic is that:
1. tdh_mem_page_aug() maps physical memory starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.
(npages corresponds to the mapping level KVM_PAGES_PER_HPAGE(level)). e.g. it can map at level 2MB, starting from the 4MB offset in a folio of order 1GB.

Quoted 2 lines

2. if split occurs, the huge 2MB mapping will be split into 4KB ones, while the underlying folio remains 1GB.

Private to shared conversion flow discussed so far [1][2][3]:
1) Preallocate maple tree entries needed for conversion
2) Split filemap range being converted to 4K pages
3) Mark KVM MMU invalidation begin for the huge page aligned range
4) Zap KVM MMU entries for the converted range
5) Update maple tree entries to carry final attributes
6) Mark KVM MMU invalidation end for huge page aligned range

Possible addition of splitting cross boundary leafs with the above flow:
1) Preallocate maple tree entries needed for conversion
2) Split filemap range being converted to 4K pages
3) Mark KVM MMU invalidation begin for the huge page aligned range
4) Split KVM MMU private boundary leafs for converted range
5) Zap KVM MMU entries for the converted range
6) Update maple tree entries to carry final attributes
7) Mark KVM MMU invalidation end for huge page aligned range

Quoted 8 lines

Note that in both the above flows KVM MMU entries will get zapped after folio is split to 4K i.e. when tdx_sept_remove_private_spte() happens folio will be split but the EPT entry level will still be 2M and the assumption of EPT entries always being subset of folios will not hold true.

I think things might be simplified if KVM TDX stack always operates on the pages without assuming ranges being covered by "folios".

Let's discuss that in v3 series
https://lore.kernel.org/all/[email protected]/

Quoted 24 lines

[1] https://lore.kernel.org/kvm/[email protected]/
[2] https://lore.kernel.org/kvm/[email protected]/
[3] https://github.com/googleprodkernel/linux-cc/blob/9ee2bd65cc9b63c871f8f49d217a7a70576a942d/virt/kvm/guest_memfd.c#L894

Quoted 20 lines

e.g. now the 0th 4KB mapping after split points to the 4MB offset in the 1GB folio, and the 1st 4KB mapping points to the 4MB+4KB offset... The mapping level after split is 4KB.

3. tdx_sept_remove_private_spte() invokes tdh_mem_page_remove() and tdh_phymem_page_wbinvd_hkid().
-The GFN is 2MB aligned and level is 2MB if split does not occur or -The GFN is 4KB aligned and level is 4KB if split has occurred. While the underlying folio remains 1GB, the folio_page_idx(folio, page) specifies the offset in the folio, and the npages corresponding to the mapping level is <= folio_nr_pages(folio).

Quoted 7 lines
Quoted 7 lines

[1] https://lore.kernel.org/all/[email protected]/

Quoted 5 lines

IIUC guest_memfd can decide
to split folios to 4K for the complete huge folio before zapping the hugepage EPT mappings. I think it's better to just round the pfn to the hugepage address based on the level they were mapped at instead of relying on the folio order.

author
#28 lore raw

RFC PATCH v2 4/23 KVM: TDX: Introduce tdx_clear_folio() to clear huge pages

In reply to Yan Zhao

After removing or reclaiming a guest private page or a control page from a TD, zero the physical page using movdir64b(), enabling the kernel to reuse the pages.

Introduce the function tdx_clear_folio() to zero out physical memory using movdir64b(), starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.

Convert tdx_clear_page() to be a helper function to facilitate the zeroing of 4KB pages.

---
RFC v2:
- Add tdx_clear_folio().
- Drop inner loop _tdx_clear_page() and move __mb() outside of the loop. (Rick)
- Use C99-style definition of variables inside a for loop.
- Note: [1] also changes tdx_clear_page(). RFC v2 is not based on [1] now.

[1] https://lore.kernel.org/all/[email protected]

RFC v1:
- split out, let tdx_clear_page() accept level.

---
 arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +16 -6
index 8eaf8431c5f1..4fabefb27135 100644
@@ -277,18 +277,21 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu)
 	vcpu->cpu = -1;
 }
 
-static void tdx_clear_page(struct page *page)
+static void tdx_clear_folio(struct folio *folio, unsigned long start_idx,
+			    unsigned long npages)
 {
 	const void *zero_page = (const void *) page_to_virt(ZERO_PAGE(0));
-	void *dest = page_to_virt(page);
-	unsigned long i;
 
 	/*
 	 * The page could have been poisoned.  MOVDIR64B also clears
 	 * the poison bit so the kernel can safely use the page again.
 	 */
-	for (i = 0; i < PAGE_SIZE; i += 64)
-		movdir64b(dest + i, zero_page);
+	for (unsigned long j = 0; j < npages; j++) {
+		void *dest = page_to_virt(folio_page(folio, start_idx + j));
+
+		for (unsigned long i = 0; i < PAGE_SIZE; i += 64)
+			movdir64b(dest + i, zero_page);
+	}
 	/*
 	 * MOVDIR64B store uses WC buffer.  Prevent following memory reads
 	 * from seeing potentially poisoned cache.
@@ -296,6 +299,13 @@ static void tdx_clear_page(struct page *page)
 	__mb();
 }
 
+static inline void tdx_clear_page(struct page *page)
+{
+	struct folio *folio = page_folio(page);
+
+	tdx_clear_folio(folio, folio_page_idx(folio, page), 1);
+}
+
 static void tdx_no_vcpus_enter_start(struct kvm *kvm)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
@@ -1736,7 +1746,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 		pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err);
 		return -EIO;
 	}
-	tdx_clear_page(page);
+	tdx_clear_folio(folio, folio_page_idx(folio, page), KVM_PAGES_PER_HPAGE(level));
 	tdx_pamt_put(page, level);
 	tdx_unpin(kvm, page);
 	return 0;
Signature · 1 line
2.43.2
#29 lore raw

Re: [RFC PATCH v2 04/23] KVM: TDX: Introduce tdx_clear_folio() to clear huge pages

Quoted 10 lines · Yan Zhao

After removing or reclaiming a guest private page or a control page from a TD, zero the physical page using movdir64b(), enabling the kernel to reuse the pages.

Introduce the function tdx_clear_folio() to zero out physical memory using movdir64b(), starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.

Convert tdx_clear_page() to be a helper function to facilitate the zeroing of 4KB pages.

I think this sentence is outdated?

Quoted 52 lines

---
RFC v2:
- Add tdx_clear_folio().
- Drop inner loop _tdx_clear_page() and move __mb() outside of the loop. (Rick)
- Use C99-style definition of variables inside a for loop.
- Note: [1] also changes tdx_clear_page(). RFC v2 is not based on [1] now.

[1] https://lore.kernel.org/all/[email protected]

RFC v1:
- split out, let tdx_clear_page() accept level.

---
  arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++------
  1 file changed, 16 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +8 -5
index 8eaf8431c5f1..4fabefb27135 100644
@@ -277,18 +277,21 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu)
  	vcpu->cpu = -1;
  }
 
-static void tdx_clear_page(struct page *page)
+static void tdx_clear_folio(struct folio *folio, unsigned long start_idx,
+			    unsigned long npages)
  {
  	const void *zero_page = (const void *) page_to_virt(ZERO_PAGE(0));
-	void *dest = page_to_virt(page);
-	unsigned long i;
 
  	/*
  	 * The page could have been poisoned.  MOVDIR64B also clears
  	 * the poison bit so the kernel can safely use the page again.
  	 */
-	for (i = 0; i < PAGE_SIZE; i += 64)
-		movdir64b(dest + i, zero_page);
+	for (unsigned long j = 0; j < npages; j++) {
+		void *dest = page_to_virt(folio_page(folio, start_idx + j));
+
+		for (unsigned long i = 0; i < PAGE_SIZE; i += 64)
+			movdir64b(dest + i, zero_page);
+	}
  	/*
  	 * MOVDIR64B store uses WC buffer.  Prevent following memory reads
  	 * from seeing potentially poisoned cache.

arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 302

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -296,6 +299,13 @@ static void tdx_clear_page(struct page *page)
  	__mb();
  }
 
+static inline void tdx_clear_page(struct page *page)

No need to tag a local static function with "inline".

arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 306

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+{
+	struct folio *folio = page_folio(page);
+
+	tdx_clear_folio(folio, folio_page_idx(folio, page), 1);

This is strange at my first thought.
And then I realized that it is to avoid unnecessary memory barrier.

No better idea so far.

Quoted 14 lines
arch/x86/kvm/vmx/tdx.c +3 -1
+}
+
  static void tdx_no_vcpus_enter_start(struct kvm *kvm)
  {
  	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
@@ -1736,7 +1746,7 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
  		pr_tdx_error(TDH_PHYMEM_PAGE_WBINVD, err);
  		return -EIO;
  	}
-	tdx_clear_page(page);
+	tdx_clear_folio(folio, folio_page_idx(folio, page), KVM_PAGES_PER_HPAGE(level));
  	tdx_pamt_put(page, level);
  	tdx_unpin(kvm, page);
  	return 0;
author
#30 lore raw

Re: [RFC PATCH v2 04/23] KVM: TDX: Introduce tdx_clear_folio() to clear huge pages

Quoted 7 lines · Binbin Wu
Quoted 4 lines · Yan Zhao

After removing or reclaiming a guest private page or a control page from a TD, zero the physical page using movdir64b(), enabling the kernel to reuse the pages.

Quoted 8 lines
Quoted 6 lines

Introduce the function tdx_clear_folio() to zero out physical memory using movdir64b(), starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.

Convert tdx_clear_page() to be a helper function to facilitate the zeroing of 4KB pages.

I think this sentence is outdated?

No? tdx_clear_page() is still invoked to clear tdr_page.

Quoted 46 lines
Quoted 46 lines

---
RFC v2:
- Add tdx_clear_folio().
- Drop inner loop _tdx_clear_page() and move __mb() outside of the loop. (Rick)
- Use C99-style definition of variables inside a for loop.
- Note: [1] also changes tdx_clear_page(). RFC v2 is not based on [1] now.

[1] https://lore.kernel.org/all/[email protected]

RFC v1:
- split out, let tdx_clear_page() accept level.

---
  arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++------
  1 file changed, 16 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +8 -5
index 8eaf8431c5f1..4fabefb27135 100644
@@ -277,18 +277,21 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu)
  	vcpu->cpu = -1;
  }
-static void tdx_clear_page(struct page *page)
+static void tdx_clear_folio(struct folio *folio, unsigned long start_idx,
+			    unsigned long npages)
  {
  	const void *zero_page = (const void *) page_to_virt(ZERO_PAGE(0));
-	void *dest = page_to_virt(page);
-	unsigned long i;
  	/*
  	 * The page could have been poisoned.  MOVDIR64B also clears
  	 * the poison bit so the kernel can safely use the page again.
  	 */
-	for (i = 0; i < PAGE_SIZE; i += 64)
-		movdir64b(dest + i, zero_page);
+	for (unsigned long j = 0; j < npages; j++) {
+		void *dest = page_to_virt(folio_page(folio, start_idx + j));
+
+		for (unsigned long i = 0; i < PAGE_SIZE; i += 64)
+			movdir64b(dest + i, zero_page);
+	}
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 301

Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +1 -0
  	/*
  	 * MOVDIR64B store uses WC buffer.  Prevent following memory reads
  	 * from seeing potentially poisoned cache.
@@ -296,6 +299,13 @@ static void tdx_clear_page(struct page *page)
  	__mb();
  }
+static inline void tdx_clear_page(struct page *page)

No need to tag a local static function with "inline".

Ok.

#31 lore raw

Re: [RFC PATCH v2 04/23] KVM: TDX: Introduce tdx_clear_folio() to clear huge pages

Quoted 7 lines · Yan Zhao
Quoted 6 lines · Binbin Wu
Quoted 4 lines · Yan Zhao

After removing or reclaiming a guest private page or a control page from a TD, zero the physical page using movdir64b(), enabling the kernel to reuse the pages.

Quoted 8 lines
Quoted 7 lines
Quoted 6 lines

Introduce the function tdx_clear_folio() to zero out physical memory using movdir64b(), starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.

Convert tdx_clear_page() to be a helper function to facilitate the zeroing of 4KB pages.

I think this sentence is outdated?

No? tdx_clear_page() is still invoked to clear tdr_page.

I didn't get the word "Convert".

Quoted 56 lines
Quoted 46 lines
Quoted 46 lines

---
RFC v2:
- Add tdx_clear_folio().
- Drop inner loop _tdx_clear_page() and move __mb() outside of the loop.

    (Rick)

- Use C99-style definition of variables inside a for loop.
- Note: [1] also changes tdx_clear_page(). RFC v2 is not based on [1] now.

[1] https://lore.kernel.org/all/[email protected]

RFC v1:
- split out, let tdx_clear_page() accept level.

---
   arch/x86/kvm/vmx/tdx.c | 22 ++++++++++++++++------
   1 file changed, 16 insertions(+), 6 deletions(-)
arch/x86/kvm/vmx/tdx.c +8 -5
index 8eaf8431c5f1..4fabefb27135 100644
@@ -277,18 +277,21 @@ static inline void tdx_disassociate_vp(struct kvm_vcpu *vcpu)
   	vcpu->cpu = -1;
   }
-static void tdx_clear_page(struct page *page)
+static void tdx_clear_folio(struct folio *folio, unsigned long start_idx,
+			    unsigned long npages)
   {
   	const void *zero_page = (const void *) page_to_virt(ZERO_PAGE(0));
-	void *dest = page_to_virt(page);
-	unsigned long i;
   	/*
   	 * The page could have been poisoned.  MOVDIR64B also clears
   	 * the poison bit so the kernel can safely use the page again.
   	 */
-	for (i = 0; i < PAGE_SIZE; i += 64)
-		movdir64b(dest + i, zero_page);
+	for (unsigned long j = 0; j < npages; j++) {
+		void *dest = page_to_virt(folio_page(folio, start_idx + j));
+
+		for (unsigned long i = 0; i < PAGE_SIZE; i += 64)
+			movdir64b(dest + i, zero_page);
+	}
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static void tdx_clear_page(struct page *page) · line 301

Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +1 -0
   	/*
   	 * MOVDIR64B store uses WC buffer.  Prevent following memory reads
   	 * from seeing potentially poisoned cache.
@@ -296,6 +299,13 @@ static void tdx_clear_page(struct page *page)
   	__mb();
   }
+static inline void tdx_clear_page(struct page *page)

No need to tag a local static function with "inline".

Ok.

author
#32 lore raw

Re: [RFC PATCH v2 04/23] KVM: TDX: Introduce tdx_clear_folio() to clear huge pages

Quoted 14 lines · Binbin Wu
Quoted 11 lines · Yan Zhao
Quoted 10 lines · Binbin Wu
Quoted 8 lines · Yan Zhao

After removing or reclaiming a guest private page or a control page from a TD, zero the physical page using movdir64b(), enabling the kernel to reuse the pages.

Introduce the function tdx_clear_folio() to zero out physical memory using movdir64b(), starting from the page at "start_idx" within a "folio" and spanning "npages" contiguous PFNs.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

Convert tdx_clear_page() to be a helper function to facilitate the zeroing of 4KB pages.

I think this sentence is outdated?

No? tdx_clear_page() is still invoked to clear tdr_page.

I didn't get the word "Convert".

Ok. I wanted to express that tdx_clear_page() now is just a helper. Will rephrase it to

"Make tdx_clear_page() to be a helper function to facilitate the zeroing of 4KB pages".

author
#33 lore raw

RFC PATCH v2 5/23 x86/tdx: Enhance tdh_phymem_page_reclaim() to support huge pages

In reply to Yan Zhao

Enhance the SEAMCALL wrapper tdh_phymem_page_reclaim() to support huge pages by introducing new parameters: "folio", "start_idx", and "npages". These parameters specify the physical memory to be reclaimed, i.e., starting from the page at "start_idx" within a folio and spanning "npages" contiguous PFNs. The specified memory must be entirely contained within a single folio. Return TDX_SW_ERROR if the size of the reclaimed memory does not match the specified size.

On the KVM side, introduce tdx_reclaim_folio() to align with and invoke the SEAMCALL wrapper tdh_phymem_page_reclaim(). The "noclear" parameter specifies whether tdx_clear_folio() should be subsequently invoked within tdx_reclaim_folio(). Additionally, provide two helper functions, tdx_reclaim_page() and tdx_reclaim_page_noclear(), to facilitate the reclaiming of 4KB pages.

---
RFC v2:
- Introduce new params "folio", "start_idx" and "npages" to wrapper tdh_phymem_page_reclaim().
- Move the checking of return size from KVM to x86/virt and return error.
- Rename tdx_reclaim_page() to tdx_reclaim_folio().
- Add two helper functions tdx_reclaim_page() tdx_reclaim_page_noclear() to faciliate the reclaiming of 4KB pages.

RFC v1:
- Rebased and split patch.

---
 arch/x86/include/asm/tdx.h  |  3 ++-
 arch/x86/kvm/vmx/tdx.c      | 27 ++++++++++++++++++---------
 arch/x86/virt/vmx/tdx/tdx.c | 12 ++++++++++--
 3 files changed, 30 insertions(+), 12 deletions(-)
arch/x86/include/asm/tdx.h +2 -1
index a125bb20a28a..f1bd74348b34 100644
@@ -189,7 +189,8 @@ u64 tdh_mng_init(struct tdx_td *td, u64 td_params, u64 *extended_err);
 u64 tdh_vp_init(struct tdx_vp *vp, u64 initial_rcx, u32 x2apicid);
 u64 tdh_vp_rd(struct tdx_vp *vp, u64 field, u64 *data);
 u64 tdh_vp_wr(struct tdx_vp *vp, u64 field, u64 data, u64 mask);
-u64 tdh_phymem_page_reclaim(struct page *page, u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size);
+u64 tdh_phymem_page_reclaim(struct folio *folio, unsigned long start_idx, unsigned long npages,
+			    u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size);
 u64 tdh_mem_track(struct tdx_td *tdr);
 u64 tdh_mem_page_remove(struct tdx_td *td, u64 gpa, u64 level, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_phymem_cache_wb(bool resume);
arch/x86/kvm/vmx/tdx.c +18 -9
index 4fabefb27135..facfe589e006 100644
@@ -327,11 +327,12 @@ static void tdx_no_vcpus_enter_stop(struct kvm *kvm)
 }
 
 /* TDH.PHYMEM.PAGE.RECLAIM is allowed only when destroying the TD. */
-static int __tdx_reclaim_page(struct page *page)
+static int tdx_reclaim_folio(struct folio *folio, unsigned long start_idx,
+			     unsigned long npages, bool noclear)
 {
 	u64 err, tdx_pt, tdx_owner, tdx_size;
 
-	err = tdh_phymem_page_reclaim(page, &tdx_pt, &tdx_owner, &tdx_size);
+	err = tdh_phymem_page_reclaim(folio, start_idx, npages, &tdx_pt, &tdx_owner, &tdx_size);
 
 	/*
 	 * No need to check for TDX_OPERAND_BUSY; all TD pages are freed
@@ -342,19 +343,25 @@ static int __tdx_reclaim_page(struct page *page)
 		pr_tdx_error_3(TDH_PHYMEM_PAGE_RECLAIM, err, tdx_pt, tdx_owner, tdx_size);
 		return -EIO;
 	}
+
+	if (!noclear)
+		tdx_clear_folio(folio, start_idx, npages);
 	return 0;
 }
 
 static int tdx_reclaim_page(struct page *page)
 {
-	int r;
+	struct folio *folio = page_folio(page);
 
-	r = __tdx_reclaim_page(page);
-	if (!r)
-		tdx_clear_page(page);
-	return r;
+	return tdx_reclaim_folio(folio, folio_page_idx(folio, page), 1, false);
 }
 
+static int tdx_reclaim_page_noclear(struct page *page)
+{
+	struct folio *folio = page_folio(page);
+
+	return tdx_reclaim_folio(folio, folio_page_idx(folio, page), 1, true);
+}
 
 /*
  * Reclaim the TD control page(s) which are crypto-protected by TDX guest's
@@ -587,7 +594,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm)
 	if (!kvm_tdx->td.tdr_page)
 		return;
 
-	if (__tdx_reclaim_page(kvm_tdx->td.tdr_page))
+	if (tdx_reclaim_page_noclear(kvm_tdx->td.tdr_page))
 		return;
 
 	/*
@@ -1932,11 +1939,13 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
 					enum pg_level level, kvm_pfn_t pfn)
 {
 	struct page *page = pfn_to_page(pfn);
+	struct folio *folio = page_folio(page);
 	int ret;
 
 	if (!is_hkid_assigned(to_kvm_tdx(kvm))) {
 		KVM_BUG_ON(!kvm->vm_dead, kvm);
-		ret = tdx_reclaim_page(page);
+		ret = tdx_reclaim_folio(folio, folio_page_idx(folio, page),
+					KVM_PAGES_PER_HPAGE(level), false);
 		if (!ret) {
 			tdx_pamt_put(page, level);
 			tdx_unpin(kvm, page);
arch/x86/virt/vmx/tdx/tdx.c +10 -2
index 64219c659844..9ed585bde062 100644
@@ -1966,19 +1966,27 @@ EXPORT_SYMBOL_GPL(tdh_vp_init);
  * So despite the names, they must be interpted specially as described by the spec. Return
  * them only for error reporting purposes.
  */
-u64 tdh_phymem_page_reclaim(struct page *page, u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size)
+u64 tdh_phymem_page_reclaim(struct folio *folio, unsigned long start_idx, unsigned long npages,
+			    u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size)
 {
+	struct page *start = folio_page(folio, start_idx);
 	struct tdx_module_args args = {
-		.rcx = page_to_phys(page),
+		.rcx = page_to_phys(start),
 	};
 	u64 ret;
 
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;
+
 	ret = seamcall_ret(TDH_PHYMEM_PAGE_RECLAIM, &args);
 
 	*tdx_pt = args.rcx;
 	*tdx_owner = args.rdx;
 	*tdx_size = args.r8;
 
+	if (npages != (1 << (*tdx_size) * PTE_SHIFT))
+		return TDX_SW_ERROR;
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(tdh_phymem_page_reclaim);
Signature · 1 line
2.43.2
#34 lore raw

Re: [RFC PATCH v2 05/23] x86/tdx: Enhance tdh_phymem_page_reclaim() to support huge pages

On 8/7/2025 5:42 PM, Yan Zhao wrote:
[...]

Quoted 20 lines
arch/x86/virt/vmx/tdx/tdx.c +5 -2
index 64219c659844..9ed585bde062 100644
@@ -1966,19 +1966,27 @@ EXPORT_SYMBOL_GPL(tdh_vp_init);
   * So despite the names, they must be interpted specially as described by the spec. Return
   * them only for error reporting purposes.
   */
-u64 tdh_phymem_page_reclaim(struct page *page, u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size)
+u64 tdh_phymem_page_reclaim(struct folio *folio, unsigned long start_idx, unsigned long npages,
+			    u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size)
  {
+	struct page *start = folio_page(folio, start_idx);
  	struct tdx_module_args args = {
-		.rcx = page_to_phys(page),
+		.rcx = page_to_phys(start),
  	};
  	u64 ret;
 
+	if (start_idx + npages > folio_nr_pages(folio))

arch/x86/virt/vmx/tdx/tdx.c EXPORT_SYMBOL_GPL(tdh_vp_init); · line 1988

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -0
+		return TDX_OPERAND_INVALID;
+
  	ret = seamcall_ret(TDH_PHYMEM_PAGE_RECLAIM, &args);
 
  	*tdx_pt = args.rcx;
  	*tdx_owner = args.rdx;
  	*tdx_size = args.r8;
 
+	if (npages != (1 << (*tdx_size) * PTE_SHIFT))
+		return TDX_SW_ERROR;

Nit:

The size check here is to  make sure the reclamation on the correct level, however, tdx_size may not be updated if some other error occurs first. Do you think it's better to check 'ret' first before returning TDX_SW_ERROR? Otherwise, the error code provided by the TDX module, which may be helpful for debugging,  will be buried under TDX_SW_ERROR.

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
+
  	return ret;
  }
  EXPORT_SYMBOL_GPL(tdh_phymem_page_reclaim);
author
#35 lore raw

Re: [RFC PATCH v2 05/23] x86/tdx: Enhance tdh_phymem_page_reclaim() to support huge pages

Quoted 34 lines · Binbin Wu

On 8/7/2025 5:42 PM, Yan Zhao wrote:
[...]

Quoted 17 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -2
index 64219c659844..9ed585bde062 100644
@@ -1966,19 +1966,27 @@ EXPORT_SYMBOL_GPL(tdh_vp_init);
   * So despite the names, they must be interpted specially as described by the spec. Return
   * them only for error reporting purposes.
   */
-u64 tdh_phymem_page_reclaim(struct page *page, u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size)
+u64 tdh_phymem_page_reclaim(struct folio *folio, unsigned long start_idx, unsigned long npages,
+			    u64 *tdx_pt, u64 *tdx_owner, u64 *tdx_size)
  {
+	struct page *start = folio_page(folio, start_idx);
  	struct tdx_module_args args = {
-		.rcx = page_to_phys(page),
+		.rcx = page_to_phys(start),
  	};

arch/x86/virt/vmx/tdx/tdx.c EXPORT_SYMBOL_GPL(tdh_vp_init); · line 1985

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +5 -0
  	u64 ret;
+	if (start_idx + npages > folio_nr_pages(folio))
+		return TDX_OPERAND_INVALID;
+
  	ret = seamcall_ret(TDH_PHYMEM_PAGE_RECLAIM, &args);
  	*tdx_pt = args.rcx;
  	*tdx_owner = args.rdx;
  	*tdx_size = args.r8;
+	if (npages != (1 << (*tdx_size) * PTE_SHIFT))
+		return TDX_SW_ERROR;

Nit:

Quoted 5 lines

The size check here is to  make sure the reclamation on the correct level, however, tdx_size may not be updated if some other error occurs first. Do you think it's better to check 'ret' first before returning TDX_SW_ERROR? Otherwise, the error code provided by the TDX module, which may be helpful for debugging,  will be buried under TDX_SW_ERROR.

Makes sense. Thanks!

I'll change it to:

	if (!ret && npages != (1 << (*tdx_size) * PTE_SHIFT))
		return TDX_SW_ERROR;
Quoted 4 lines
Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -0
  	return ret;
  }
  EXPORT_SYMBOL_GPL(tdh_phymem_page_reclaim);
author
#36 lore raw

RFC PATCH v2 6/23 KVM: TDX: Do not hold page refcount on private guest pages

In reply to Yan Zhao

To enable guest_memfd to support in-place conversion between shared and private memory [1], TDX is required not to hold refcount of the private pages allocated from guest_memfd.

Due to that a folio only has a single refcount and the need to reliably determine unexpected reference when converting any shared part to private, guest_memfd [1] does not permit shared memory to be huge [2]. Consequently, it must split private huge pages into 4KB shared pages. However, since guest_memfd cannot distinguish between the speculative/transient refcounts and the intentional refcount for TDX on private pages[3], failing to release private page refcount in TDX could cause guest_memfd to indefinitely wait on decreasing the refcount for the splitting.

Under normal conditions, not holding an extra page refcount in TDX is safe because guest_memfd ensures pages are retained until its invalidation notification to KVM MMU is completed. However, if there're bugs in KVM/TDX module, not holding an extra refcount when a page is mapped in S-EPT could result in a page being released from guest_memfd while still mapped in the S-EPT.

Several approaches were considered to address this issue, including
- Attempting to modify the KVM unmap operation to return a failure, which was deemed too complex and potentially incorrect [4].
- Increasing the folio reference count only upon S-EPT zapping failure [5].
- Use page flags or page_ext to indicate a page is still used by TDX [6], which does not work for HVO (HugeTLB Vmemmap Optimization).
- Setting HWPOISON bit or leveraging folio_set_hugetlb_hwpoison() [7].

Due to the complexity or inappropriateness of these approaches, and the fact that S-EPT zapping failure is currently only possible when there are bugs in the KVM or TDX module, which is very rare in a production kernel, a straightforward approach of simply not holding the page reference count in TDX was chosen [8].

When S-EPT zapping errors occur, KVM_BUG_ON() is invoked to kick off all vCPUs and mark the VM as dead. Although there is a potential window that a private page mapped in the S-EPT could be reallocated and used outside the VM, the loud warning from KVM_BUG_ON() should provide sufficient debug information. To be robust against bugs, the user can enable panic_on_warn as normal.

---
RFC v2:
- new in RFC v2.
- Rebased on DPAMT and shutdown optimization.

---
 arch/x86/kvm/vmx/tdx.c | 28 ++++------------------------
 1 file changed, 4 insertions(+), 24 deletions(-)
arch/x86/kvm/vmx/tdx.c +4 -24
index facfe589e006..376287a2ddf4 100644
@@ -1600,11 +1600,6 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)
 	td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa);
 }
 
-static void tdx_unpin(struct kvm *kvm, struct page *page)
-{
-	put_page(page);
-}
-
 static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
 			    enum pg_level level, struct page *page)
 {
@@ -1617,14 +1612,11 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
 
 	err = tdh_mem_page_aug(&kvm_tdx->td, gpa, tdx_level, folio,
 			       folio_page_idx(folio, page), &entry, &level_state);
-	if (unlikely(tdx_operand_busy(err))) {
-		tdx_unpin(kvm, page);
+	if (unlikely(tdx_operand_busy(err)))
 		return -EBUSY;
-	}
 
 	if (KVM_BUG_ON(err, kvm)) {
 		pr_tdx_error_2(TDH_MEM_PAGE_AUG, err, entry, level_state);
-		tdx_unpin(kvm, page);
 		return -EIO;
 	}
 
@@ -1679,16 +1671,6 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,
 	if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
 		return -EINVAL;
 
-	/*
-	 * Because guest_memfd doesn't support page migration with
-	 * a_ops->migrate_folio (yet), no callback is triggered for KVM on page
-	 * migration.  Until guest_memfd supports page migration, prevent page
-	 * migration.
-	 * TODO: Once guest_memfd introduces callback on page migration,
-	 * implement it and remove get_page/put_page().
-	 */
-	get_page(page);
-
 	/*
 	 * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching
 	 * barrier in tdx_td_finalize().
@@ -1755,7 +1737,6 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 	}
 	tdx_clear_folio(folio, folio_page_idx(folio, page), KVM_PAGES_PER_HPAGE(level));
 	tdx_pamt_put(page, level);
-	tdx_unpin(kvm, page);
 	return 0;
 }
 
@@ -1845,7 +1826,6 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,
 	    !KVM_BUG_ON(!atomic64_read(&kvm_tdx->nr_premapped), kvm)) {
 		atomic64_dec(&kvm_tdx->nr_premapped);
 		tdx_pamt_put(page, level);
-		tdx_unpin(kvm, page);
 		return 0;
 	}
 
@@ -1944,12 +1924,12 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
 
 	if (!is_hkid_assigned(to_kvm_tdx(kvm))) {
 		KVM_BUG_ON(!kvm->vm_dead, kvm);
+
 		ret = tdx_reclaim_folio(folio, folio_page_idx(folio, page),
 					KVM_PAGES_PER_HPAGE(level), false);
-		if (!ret) {
+		if (!ret)
 			tdx_pamt_put(page, level);
-			tdx_unpin(kvm, page);
-		}
+
 		return ret;
 	}
 
Signature · 1 line
2.43.2
author
#37 lore raw

RFC PATCH v2 7/23 KVM: x86/mmu: Disallow page merging (huge page adjustment) for mirror root

In reply to Yan Zhao

From: "Edgecombe, Rick P" <[email protected]>

Disallow page merging (huge page adjustment) for the mirror root by utilizing disallowed_hugepage_adjust().

Make the mirror root check asymmetric with NX huge pages and not to litter the generic MMU code:

Invoke disallowed_hugepage_adjust() in kvm_tdp_mmu_map() when necessary, specifically when KVM has mirrored TDP or the NX huge page workaround is enabled.

Check and reduce the goal_level of a fault internally in disallowed_hugepage_adjust() when the fault is for a mirror root and there's a shadow present non-leaf entry at the original goal_level.

---
RFC v2:
- Check is_mirror_sp() in disallowed_hugepage_adjust() instead of passing in an is_mirror arg. (Rick)
- Check kvm_has_mirrored_tdp() in kvm_tdp_mmu_map() to determine whether to invoke disallowed_hugepage_adjust(). (Rick)

RFC v1:
- new patch

---
 arch/x86/kvm/mmu/mmu.c     | 3 ++-
 arch/x86/kvm/mmu/tdp_mmu.c | 4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/mmu.c +2 -1
index 3f76415cec71..9182192daa3a 100644
@@ -3412,7 +3412,8 @@ void disallowed_hugepage_adjust(struct kvm_page_fault *fault, u64 spte, int cur_
 	    cur_level == fault->goal_level &&
 	    is_shadow_present_pte(spte) &&
 	    !is_large_pte(spte) &&
-	    spte_to_child_sp(spte)->nx_huge_page_disallowed) {
+	    ((spte_to_child_sp(spte)->nx_huge_page_disallowed) ||
+	     is_mirror_sp(spte_to_child_sp(spte)))) {
 		/*
 		 * A small SPTE exists for this pfn, but FNAME(fetch),
 		 * direct_map(), or kvm_tdp_mmu_map() would like to create a
arch/x86/kvm/mmu/tdp_mmu.c +3 -1
index bb95c95f6531..f9a054754544 100644
@@ -1243,6 +1243,8 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 	struct tdp_iter iter;
 	struct kvm_mmu_page *sp;
 	int ret = RET_PF_RETRY;
+	bool hugepage_adjust_disallowed = fault->nx_huge_page_workaround_enabled ||
+					  kvm_has_mirrored_tdp(kvm);
 
 	kvm_mmu_hugepage_adjust(vcpu, fault);
 
@@ -1253,7 +1255,7 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
 	for_each_tdp_pte(iter, kvm, root, fault->gfn, fault->gfn + 1) {
 		int r;
 
-		if (fault->nx_huge_page_workaround_enabled)
+		if (hugepage_adjust_disallowed)
 			disallowed_hugepage_adjust(fault, iter.old_spte, iter.level);
 
 		/*
Signature · 1 line
2.43.2
author
#38 lore raw

RFC PATCH v2 8/23 KVM: x86/tdp_mmu: Alloc external_spt page for mirror page table splitting

In reply to Yan Zhao

From: Isaku Yamahata <[email protected]>

Enhance tdp_mmu_alloc_sp_split() to allocate a page for sp->external_spt, i.e., the external page table page, for splitting the mirror page table.

---
RFC v2:
- NO change.

RFC v1:
- Rebased and simplified the code.

---
 arch/x86/kvm/mmu/tdp_mmu.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
arch/x86/kvm/mmu/tdp_mmu.c +13 -2
index f9a054754544..46b9f276bb6d 100644
@@ -324,6 +324,8 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,
 				u64 old_spte, u64 new_spte, int level,
 				bool shared);
 
+static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror);
+
 static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
 	kvm_account_pgtable_pages((void *)sp->spt, +1);
@@ -1475,7 +1477,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm,
 	return spte_set;
 }
 
-static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(void)
+static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror)
 {
 	struct kvm_mmu_page *sp;
 
@@ -1489,6 +1491,15 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(void)
 		return NULL;
 	}
 
+	if (mirror) {
+		sp->external_spt = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
+		if (!sp->external_spt) {
+			free_page((unsigned long)sp->spt);
+			kmem_cache_free(mmu_page_header_cache, sp);
+			return NULL;
+		}
+	}
+
 	return sp;
 }
 
@@ -1568,7 +1579,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
 			else
 				write_unlock(&kvm->mmu_lock);
 
-			sp = tdp_mmu_alloc_sp_for_split();
+			sp = tdp_mmu_alloc_sp_for_split(is_mirror_sp(root));
 
 			if (shared)
 				read_lock(&kvm->mmu_lock);
Signature · 1 line
2.43.2
#39 lore raw

Re: [RFC PATCH v2 08/23] KVM: x86/tdp_mmu: Alloc external_spt page for mirror page table splitting

Quoted 3 lines · Yan Zhao

From: Isaku Yamahata <[email protected]>

Enhance tdp_mmu_alloc_sp_split() to allocate a page for sp->external_spt,

	  ^
	  tdp_mmu_alloc_sp_for_split()

[...]

arch/x86/kvm/mmu/tdp_mmu.c static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, · line 328

Quoted 2 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
+static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror);
+

It doesn't seem you need such declaration in _this_ patch. If any later patch needs it, then perhaps it's better to do in that patch.

Quoted 37 lines

static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) {

arch/x86/kvm/mmu/tdp_mmu.c +11 -2
 	kvm_account_pgtable_pages((void *)sp->spt, +1);
@@ -1475,7 +1477,7 @@ bool kvm_tdp_mmu_wrprot_slot(struct kvm *kvm,
 	return spte_set;
 }
 
-static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(void)
+static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror)
 {
 	struct kvm_mmu_page *sp;
 
@@ -1489,6 +1491,15 @@ static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(void)
 		return NULL;
 	}
 
+	if (mirror) {
+		sp->external_spt = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT);
+		if (!sp->external_spt) {
+			free_page((unsigned long)sp->spt);
+			kmem_cache_free(mmu_page_header_cache, sp);
+			return NULL;
+		}
+	}
+
 	return sp;
 }
 
@@ -1568,7 +1579,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
 			else
 				write_unlock(&kvm->mmu_lock);
 
-			sp = tdp_mmu_alloc_sp_for_split();
+			sp = tdp_mmu_alloc_sp_for_split(is_mirror_sp(root));
 
 			if (shared)
 				read_lock(&kvm->mmu_lock);
author
#40 lore raw

Re: [RFC PATCH v2 08/23] KVM: x86/tdp_mmu: Alloc external_spt page for mirror page table splitting

Quoted 6 lines · Kai
Quoted 3 lines · Yan Zhao

From: Isaku Yamahata <[email protected]>

Enhance tdp_mmu_alloc_sp_split() to allocate a page for sp->external_spt,

	  ^
	  tdp_mmu_alloc_sp_for_split()

Right.

Quoted 7 lines

[...]

arch/x86/kvm/mmu/tdp_mmu.c static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn, · line 328

Quoted 2 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
+static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror);
+

It doesn't seem you need such declaration in _this_ patch. If any later patch needs it, then perhaps it's better to do in that patch.

Thanks!
I'll drop this declaration. It's no longer needed for v2, because its caller is now tdp_mmu_split_huge_pages_root(), which is later than tdp_mmu_alloc_sp_for_split().

author
#41 lore raw

RFC PATCH v2 9/23 KVM: x86/tdp_mmu: Add split_external_spt hook called during write mmu_lock

In reply to Yan Zhao

Introduce the split_external_spt hook and call it within tdp_mmu_set_spte() for the mirror page table.

tdp_mmu_set_spte() is invoked for SPTE transitions under write mmu_lock. For the mirror page table, in addition to the valid transitions from a shadow-present entry to !shadow-present entry, introduce a new valid transition case for splitting and propagate the transition to the external page table via the hook split_external_spt.

---
RFC v2:
- Removed the KVM_BUG_ON() in split_external_spt(). (Rick)
- Add a comment for the KVM_BUG_ON() in tdp_mmu_set_spte(). (Rick)
- Use kvm_x86_call() instead of static_call(). (Binbin)

RFC v1:
- Split patch.
- Dropped invoking hook zap_private_spte and kvm_flush_remote_tlbs() in KVM MMU core.

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  4 ++++
 arch/x86/kvm/mmu/tdp_mmu.c         | 29 +++++++++++++++++++++++++----
 3 files changed, 30 insertions(+), 4 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 18a5c3119e1a..7653a45ad5b2 100644
@@ -98,6 +98,7 @@ KVM_X86_OP_OPTIONAL(link_external_spt)
 KVM_X86_OP_OPTIONAL(set_external_spte)
 KVM_X86_OP_OPTIONAL(free_external_spt)
 KVM_X86_OP_OPTIONAL(remove_external_spte)
+KVM_X86_OP_OPTIONAL(split_external_spt)
 KVM_X86_OP(has_wbinvd_exit)
 KVM_X86_OP(get_l2_tsc_offset)
 KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +4 -0
index 823d1aeef2a8..e431ce0e3180 100644
@@ -1839,6 +1839,10 @@ struct kvm_x86_ops {
 	int (*remove_external_spte)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
 				    kvm_pfn_t pfn_for_gfn);
 
+	/* Split the external page table into smaller page tables */
+	int (*split_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
+				  void *external_spt);
+
 	bool (*has_wbinvd_exit)(void);
 
 	u64 (*get_l2_tsc_offset)(struct kvm_vcpu *vcpu);
arch/x86/kvm/mmu/tdp_mmu.c +25 -4
index 46b9f276bb6d..a2c6e6e4773f 100644
@@ -325,6 +325,7 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,
 				bool shared);
 
 static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror);
+static void *get_external_spt(gfn_t gfn, u64 new_spte, int level);
 
 static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp)
 {
@@ -384,6 +385,18 @@ static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
 	KVM_BUG_ON(ret, kvm);
 }
 
+static int split_external_spt(struct kvm *kvm, gfn_t gfn, u64 old_spte,
+			      u64 new_spte, int level)
+{
+	void *external_spt = get_external_spt(gfn, new_spte, level);
+	int ret;
+
+	KVM_BUG_ON(!external_spt, kvm);
+
+	ret = kvm_x86_call(split_external_spt)(kvm, gfn, level, external_spt);
+
+	return ret;
+}
 /**
  * handle_removed_pt() - handle a page table removed from the TDP structure
  *
@@ -765,12 +778,20 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
 	handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false);
 
 	/*
-	 * Users that do non-atomic setting of PTEs don't operate on mirror
-	 * roots, so don't handle it and bug the VM if it's seen.
+	 * Propagate changes of SPTE to the external page table under write
+	 * mmu_lock.
+	 * Current valid transitions:
+	 * - present leaf to !present.
+	 * - present non-leaf to !present.
+	 * - present leaf to present non-leaf (splitting)
 	 */
 	if (is_mirror_sptep(sptep)) {
-		KVM_BUG_ON(is_shadow_present_pte(new_spte), kvm);
-		remove_external_spte(kvm, gfn, old_spte, level);
+		if (!is_shadow_present_pte(new_spte))
+			remove_external_spte(kvm, gfn, old_spte, level);
+		else if (is_last_spte(old_spte, level) && !is_last_spte(new_spte, level))
+			split_external_spt(kvm, gfn, old_spte, new_spte, level);
+		else
+			KVM_BUG_ON(1, kvm);
 	}
 
 	return old_spte;
Signature · 1 line
2.43.2
#42 lore raw

Re: [RFC PATCH v2 09/23] KVM: x86/tdp_mmu: Add split_external_spt hook called during write mmu_lock

Quoted 2 lines · Yan Zhao

Introduce the split_external_spt hook and call it within tdp_mmu_set_spte() for the mirror page table.

Nit: I think you need to use split_external_spt() since it's a function, even you already mentioned it is a hook.

Quoted 6 lines

tdp_mmu_set_spte() is invoked for SPTE transitions under write mmu_lock. For the mirror page table, in addition to the valid transitions from a shadow-present entry to !shadow-present entry, introduce a new valid transition case for splitting and propagate the transition to the external page table via the hook split_external_spt.

Ditto: split_external_spt()

[...]

Quoted 2 lines

static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror);
+static void *get_external_spt(gfn_t gfn, u64 new_spte, int level);

Is it possible to get rid of such declarations, e.g., by ...

Quoted 9 lines

static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) {

arch/x86/kvm/mmu/tdp_mmu.c +2 -0
@@ -384,6 +385,18 @@ static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
 	KVM_BUG_ON(ret, kvm);
 }
 
+static int split_external_spt(struct kvm *kvm, gfn_t gfn, u64 old_spte,
+			      u64 new_spte, int level)

arch/x86/kvm/mmu/tdp_mmu.c static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte, · line 399

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10 -0
+{
+	void *external_spt = get_external_spt(gfn, new_spte, level);
+	int ret;
+
+	KVM_BUG_ON(!external_spt, kvm);
+
+	ret = kvm_x86_call(split_external_spt)(kvm, gfn, level, external_spt);
+
+	return ret;
+}

... moving split_external_spt() somewhere else, e.g., after set_external_spte_present() (which calls get_external_spt())?

Since ...

Quoted 17 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -2
 /**
  * handle_removed_pt() - handle a page table removed from the TDP structure
  *
@@ -765,12 +778,20 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
 	handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false);
 
 	/*
-	 * Users that do non-atomic setting of PTEs don't operate on mirror
-	 * roots, so don't handle it and bug the VM if it's seen.
+	 * Propagate changes of SPTE to the external page table under write
+	 * mmu_lock.
+	 * Current valid transitions:
+	 * - present leaf to !present.
+	 * - present non-leaf to !present.
+	 * - present leaf to present non-leaf (splitting)
 	 */
 	if (is_mirror_sptep(sptep)) {

arch/x86/kvm/mmu/tdp_mmu.c static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, · line 796

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -2
-		KVM_BUG_ON(is_shadow_present_pte(new_spte), kvm);
-		remove_external_spte(kvm, gfn, old_spte, level);
+		if (!is_shadow_present_pte(new_spte))
+			remove_external_spte(kvm, gfn, old_spte, level);
+		else if (is_last_spte(old_spte, level) && !is_last_spte(new_spte, level))
+			split_external_spt(kvm, gfn, old_spte, new_spte, level);
+		else
+			KVM_BUG_ON(1, kvm);
 	}
 

... split_external_spt() is only called here in tdp_mmu_set_spte() which is way after set_external_spte_present() AFAICT.

author
#43 lore raw

Re: [RFC PATCH v2 09/23] KVM: x86/tdp_mmu: Add split_external_spt hook called during write mmu_lock

Quoted 6 lines · Kai
Quoted 2 lines · Yan Zhao

Introduce the split_external_spt hook and call it within tdp_mmu_set_spte() for the mirror page table.

Nit: I think you need to use split_external_spt() since it's a function, even you already mentioned it is a hook.

Makes sense.

Quoted 7 lines
Quoted 5 lines

tdp_mmu_set_spte() is invoked for SPTE transitions under write mmu_lock. For the mirror page table, in addition to the valid transitions from a shadow-present entry to !shadow-present entry, introduce a new valid transition case for splitting and propagate the transition to the external page table via the hook split_external_spt.

Ditto: split_external_spt()

Will update.

Quoted 6 lines

[...]

Quoted 2 lines

static struct kvm_mmu_page *tdp_mmu_alloc_sp_for_split(bool mirror);
+static void *get_external_spt(gfn_t gfn, u64 new_spte, int level);

Is it possible to get rid of such declarations, e.g., by ...

I think so.

Will drop this declaration by moving split_external_spt() after get_external_spt() but before set_external_spte_present() and tdp_mmu_set_spte().

Thanks for this suggestion.

Quoted 54 lines
Quoted 8 lines

static void tdp_account_mmu_page(struct kvm *kvm, struct kvm_mmu_page *sp) {

arch/x86/kvm/mmu/tdp_mmu.c +2 -0
@@ -384,6 +385,18 @@ static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte,
 	KVM_BUG_ON(ret, kvm);
 }
 
+static int split_external_spt(struct kvm *kvm, gfn_t gfn, u64 old_spte,
+			      u64 new_spte, int level)

arch/x86/kvm/mmu/tdp_mmu.c static void remove_external_spte(struct kvm *kvm, gfn_t gfn, u64 old_spte, · line 399

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10 -0
+{
+	void *external_spt = get_external_spt(gfn, new_spte, level);
+	int ret;
+
+	KVM_BUG_ON(!external_spt, kvm);
+
+	ret = kvm_x86_call(split_external_spt)(kvm, gfn, level, external_spt);
+
+	return ret;
+}

... moving split_external_spt() somewhere else, e.g., after set_external_spte_present() (which calls get_external_spt())?

Since ...

Quoted 17 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -2
 /**
  * handle_removed_pt() - handle a page table removed from the TDP structure
  *
@@ -765,12 +778,20 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
 	handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false);
 
 	/*
-	 * Users that do non-atomic setting of PTEs don't operate on mirror
-	 * roots, so don't handle it and bug the VM if it's seen.
+	 * Propagate changes of SPTE to the external page table under write
+	 * mmu_lock.
+	 * Current valid transitions:
+	 * - present leaf to !present.
+	 * - present non-leaf to !present.
+	 * - present leaf to present non-leaf (splitting)
 	 */
 	if (is_mirror_sptep(sptep)) {

arch/x86/kvm/mmu/tdp_mmu.c static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, · line 796

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -2
-		KVM_BUG_ON(is_shadow_present_pte(new_spte), kvm);
-		remove_external_spte(kvm, gfn, old_spte, level);
+		if (!is_shadow_present_pte(new_spte))
+			remove_external_spte(kvm, gfn, old_spte, level);
+		else if (is_last_spte(old_spte, level) && !is_last_spte(new_spte, level))
+			split_external_spt(kvm, gfn, old_spte, new_spte, level);
+		else
+			KVM_BUG_ON(1, kvm);
 	}
 

... split_external_spt() is only called here in tdp_mmu_set_spte() which is way after set_external_spte_present() AFAICT.

#44 lore raw

Re: [RFC PATCH v2 09/23] KVM: x86/tdp_mmu: Add split_external_spt hook called during write mmu_lock

In reply to Yan Zhao

On 8/7/2025 5:43 PM, Yan Zhao wrote:
[...]

Quoted 4 lines

/**
* handle_removed_pt() - handle a page table removed from the TDP structure *

arch/x86/kvm/mmu/tdp_mmu.c static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, · line 785

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +5 -2
@@ -765,12 +778,20 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
  	handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false);
 
  	/*
-	 * Users that do non-atomic setting of PTEs don't operate on mirror
-	 * roots, so don't handle it and bug the VM if it's seen.
+	 * Propagate changes of SPTE to the external page table under write
+	 * mmu_lock.
+	 * Current valid transitions:
+	 * - present leaf to !present.
+	 * - present non-leaf to !present.

Nit:
Maybe add a small note to limit the scenario, such as "after releasing the HKID" or "during the TD teardown"?

Quoted 14 lines
arch/x86/kvm/mmu/tdp_mmu.c +7 -2
+	 * - present leaf to present non-leaf (splitting)
  	 */
  	if (is_mirror_sptep(sptep)) {
-		KVM_BUG_ON(is_shadow_present_pte(new_spte), kvm);
-		remove_external_spte(kvm, gfn, old_spte, level);
+		if (!is_shadow_present_pte(new_spte))
+			remove_external_spte(kvm, gfn, old_spte, level);
+		else if (is_last_spte(old_spte, level) && !is_last_spte(new_spte, level))
+			split_external_spt(kvm, gfn, old_spte, new_spte, level);
+		else
+			KVM_BUG_ON(1, kvm);
  	}
 
  	return old_spte;
author
#45 lore raw

Re: [RFC PATCH v2 09/23] KVM: x86/tdp_mmu: Add split_external_spt hook called during write mmu_lock

Quoted 13 lines · Binbin Wu

On 8/7/2025 5:43 PM, Yan Zhao wrote:
[...]

Quoted 9 lines

/**
* handle_removed_pt() - handle a page table removed from the TDP structure *

arch/x86/kvm/mmu/tdp_mmu.c +1 -2
@@ -765,12 +778,20 @@ static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep,
  	handle_changed_spte(kvm, as_id, gfn, old_spte, new_spte, level, false);
  	/*
-	 * Users that do non-atomic setting of PTEs don't operate on mirror
-	 * roots, so don't handle it and bug the VM if it's seen.
+	 * Propagate changes of SPTE to the external page table under write
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c static u64 tdp_mmu_set_spte(struct kvm *kvm, int as_id, tdp_ptep_t sptep, · line 785

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+	 * mmu_lock.
+	 * Current valid transitions:
+	 * - present leaf to !present.
+	 * - present non-leaf to !present.

Nit:
Maybe add a small note to limit the scenario, such as "after releasing the HKID" or "during the TD teardown"?

I'm not sure if we need that level of detail.
e.g., for case "present leaf to !present", it's before releasing the HKID w/o patch [1], but can be after the HKID w/ patch [1].

[1] https://lore.kernel.org/all/[email protected]/

Quoted 14 lines
Quoted 13 lines
arch/x86/kvm/mmu/tdp_mmu.c +7 -2
+	 * - present leaf to present non-leaf (splitting)
  	 */
  	if (is_mirror_sptep(sptep)) {
-		KVM_BUG_ON(is_shadow_present_pte(new_spte), kvm);
-		remove_external_spte(kvm, gfn, old_spte, level);
+		if (!is_shadow_present_pte(new_spte))
+			remove_external_spte(kvm, gfn, old_spte, level);
+		else if (is_last_spte(old_spte, level) && !is_last_spte(new_spte, level))
+			split_external_spt(kvm, gfn, old_spte, new_spte, level);
+		else
+			KVM_BUG_ON(1, kvm);
  	}
  	return old_spte;
author
#46 lore raw

RFC PATCH v2 10/23 KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

In reply to Yan Zhao

Implement the split_external_spt hook to enable huge page splitting for TDX when kvm->mmu_lock is held for writing.

Invoke tdh_mem_range_block(), tdh_mem_track(), kicking off vCPUs, tdh_mem_page_demote() in sequence. All operations are performed under kvm->mmu_lock held for writing, similar to those in page removal.

Even with kvm->mmu_lock held for writing, tdh_mem_page_demote() may still contend with tdh_vp_enter() and potentially with the guest's S-EPT entry operations. Therefore, kick off other vCPUs and prevent tdh_vp_enter() from being called on them to ensure success on the second attempt. Use KVM_BUG_ON() for any other unexpected errors.

---
RFC v2:
- Split out the code to handle the error TDX_INTERRUPTED_RESTARTABLE.
- Rebased to 6.16.0-rc6 (the way of defining TDX hook changes).

RFC v1:
- Split patch for exclusive mmu_lock only,
- Invoke tdx_sept_zap_private_spte() and tdx_track() for splitting.
- Handled busy error of tdh_mem_page_demote() by kicking off vCPUs.

---
 arch/x86/kvm/vmx/tdx.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
arch/x86/kvm/vmx/tdx.c +45 -0
index 376287a2ddf4..8a60ba5b6595 100644
@@ -1915,6 +1915,50 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,
 	return 0;
 }
 
+static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
+					enum pg_level level, struct page *page)
+{
+	int tdx_level = pg_level_to_tdx_sept_level(level);
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	gpa_t gpa = gfn_to_gpa(gfn);
+	u64 err, entry, level_state;
+
+	err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+				  &entry, &level_state);
+
+	if (unlikely(tdx_operand_busy(err))) {
+		tdx_no_vcpus_enter_start(kvm);
+		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+					  &entry, &level_state);
+		tdx_no_vcpus_enter_stop(kvm);
+	}
+
+	if (KVM_BUG_ON(err, kvm)) {
+		pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
+		return -EIO;
+	}
+	return 0;
+}
+
+static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,
+				      void *private_spt)
+{
+	struct page *page = virt_to_page(private_spt);
+	int ret;
+
+	if (KVM_BUG_ON(to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE ||
+		       level != PG_LEVEL_2M, kvm))
+		return -EINVAL;
+
+	ret = tdx_sept_zap_private_spte(kvm, gfn, level, page);
+	if (ret <= 0)
+		return ret;
+
+	tdx_track(kvm);
+
+	return tdx_spte_demote_private_spte(kvm, gfn, level, page);
+}
+
 static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
 					enum pg_level level, kvm_pfn_t pfn)
 {
@@ -3668,5 +3712,6 @@ void __init tdx_hardware_setup(void)
 	vt_x86_ops.set_external_spte = tdx_sept_set_private_spte;
 	vt_x86_ops.free_external_spt = tdx_sept_free_private_spt;
 	vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte;
+	vt_x86_ops.split_external_spt = tdx_sept_split_private_spt;
 	vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt;
 }
Signature · 1 line
2.43.2
#47 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 7 lines · Yan Zhao

Implement the split_external_spt hook to enable huge page splitting for TDX when kvm->mmu_lock is held for writing.

Invoke tdh_mem_range_block(), tdh_mem_track(), kicking off vCPUs, tdh_mem_page_demote() in sequence. All operations are performed under kvm->mmu_lock held for writing, similar to those in page removal.

Quoted 5 lines

Even with kvm->mmu_lock held for writing, tdh_mem_page_demote() may still contend with tdh_vp_enter() and potentially with the guest's S-EPT entry operations. Therefore, kick off other vCPUs and prevent tdh_vp_enter() from being called on them to ensure success on the second attempt. Use KVM_BUG_ON() for any other unexpected errors.

I thought we also need to do UNBLOCK after DEMOTE, but it turns out we don't need to. Maybe we can call this out.

[...]

Quoted 27 lines
arch/x86/kvm/vmx/tdx.c +26 -0
+static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
+					enum pg_level level, struct page *page)
+{
+	int tdx_level = pg_level_to_tdx_sept_level(level);
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	gpa_t gpa = gfn_to_gpa(gfn);
+	u64 err, entry, level_state;
+
+	err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+				  &entry, &level_state);
+
+	if (unlikely(tdx_operand_busy(err))) {
+		tdx_no_vcpus_enter_start(kvm);
+		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+					  &entry, &level_state);
+		tdx_no_vcpus_enter_stop(kvm);
+	}
+
+	if (KVM_BUG_ON(err, kvm)) {
+		pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
+		return -EIO;
+	}
+	return 0;
+}
+
+static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1953

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+				      void *private_spt)
+{
+	struct page *page = virt_to_page(private_spt);
+	int ret;
+
+	if (KVM_BUG_ON(to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE ||
+		       level != PG_LEVEL_2M, kvm))
+		return -EINVAL;
+
+	ret = tdx_sept_zap_private_spte(kvm, gfn, level, page);

I don't quite follow why you pass 'private_spt' to tdx_sept_zap_private_spte(), but it doesn't matter anymore since it's gone in Sean's latest tree.

author
#48 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 8 lines · Kai
Quoted 7 lines · Yan Zhao

Implement the split_external_spt hook to enable huge page splitting for TDX when kvm->mmu_lock is held for writing.

Invoke tdh_mem_range_block(), tdh_mem_track(), kicking off vCPUs, tdh_mem_page_demote() in sequence. All operations are performed under kvm->mmu_lock held for writing, similar to those in page removal.

Quoted 8 lines
Quoted 5 lines

Even with kvm->mmu_lock held for writing, tdh_mem_page_demote() may still contend with tdh_vp_enter() and potentially with the guest's S-EPT entry operations. Therefore, kick off other vCPUs and prevent tdh_vp_enter() from being called on them to ensure success on the second attempt. Use KVM_BUG_ON() for any other unexpected errors.

I thought we also need to do UNBLOCK after DEMOTE, but it turns out we don't need to.

Yes, the BLOCK operates on PG_LEVEL_2M, and a successful DEMOTE updates the SEPT non-leaf 2MB entry to point to the newly added page table page with RWX permission, so there's no need to do UNBLOCK on success.

The purpose of BLOCK + TRACK + kick off vCPUs is to ensure all vCPUs must find the old huge guest page is no longer mapped in the SEPT.

Quoted 1 line

Maybe we can call this out.

Will do.

Quoted 31 lines
Quoted 31 lines
arch/x86/kvm/vmx/tdx.c +31 -0
+static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
+					enum pg_level level, struct page *page)
+{
+	int tdx_level = pg_level_to_tdx_sept_level(level);
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	gpa_t gpa = gfn_to_gpa(gfn);
+	u64 err, entry, level_state;
+
+	err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+				  &entry, &level_state);
+
+	if (unlikely(tdx_operand_busy(err))) {
+		tdx_no_vcpus_enter_start(kvm);
+		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+					  &entry, &level_state);
+		tdx_no_vcpus_enter_stop(kvm);
+	}
+
+	if (KVM_BUG_ON(err, kvm)) {
+		pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
+		return -EIO;
+	}
+	return 0;
+}
+
+static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,
+				      void *private_spt)
+{
+	struct page *page = virt_to_page(private_spt);
+	int ret;
+
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1953

Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5 -0
+	if (KVM_BUG_ON(to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE ||
+		       level != PG_LEVEL_2M, kvm))
+		return -EINVAL;
+
+	ret = tdx_sept_zap_private_spte(kvm, gfn, level, page);

I don't quite follow why you pass 'private_spt' to tdx_sept_zap_private_spte(),

Simply because tdx_sept_zap_private_spte() requires a "page", which is actually not used by tdx_sept_zap_private_spte() in the split path.

Quoted 2 lines

but it doesn't matter anymore since it's gone
in Sean's latest tree.

Right.

#49 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 3 lines · Yan Zhao
Quoted 2 lines · Kai
Quoted 1 line · Yan Zhao

Implement the split_external_spt hook to enable huge page splitting for

Nit:
split_external_spt(), similar as Kai mentioned in patch 9.

Quoted 68 lines
Quoted 6 lines
Quoted 6 lines

TDX when kvm->mmu_lock is held for writing.

Invoke tdh_mem_range_block(), tdh_mem_track(), kicking off vCPUs, tdh_mem_page_demote() in sequence. All operations are performed under kvm->mmu_lock held for writing, similar to those in page removal.

Quoted 7 lines
Quoted 5 lines

Even with kvm->mmu_lock held for writing, tdh_mem_page_demote() may still contend with tdh_vp_enter() and potentially with the guest's S-EPT entry operations. Therefore, kick off other vCPUs and prevent tdh_vp_enter() from being called on them to ensure success on the second attempt. Use KVM_BUG_ON() for any other unexpected errors.

I thought we also need to do UNBLOCK after DEMOTE, but it turns out we don't need to.

Yes, the BLOCK operates on PG_LEVEL_2M, and a successful DEMOTE updates the SEPT non-leaf 2MB entry to point to the newly added page table page with RWX permission, so there's no need to do UNBLOCK on success.

The purpose of BLOCK + TRACK + kick off vCPUs is to ensure all vCPUs must find the old huge guest page is no longer mapped in the SEPT.

Quoted 1 line

Maybe we can call this out.

Will do.

Quoted 30 lines
Quoted 30 lines
arch/x86/kvm/vmx/tdx.c +30 -0
+static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
+					enum pg_level level, struct page *page)
+{
+	int tdx_level = pg_level_to_tdx_sept_level(level);
+	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+	gpa_t gpa = gfn_to_gpa(gfn);
+	u64 err, entry, level_state;
+
+	err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+				  &entry, &level_state);
+
+	if (unlikely(tdx_operand_busy(err))) {
+		tdx_no_vcpus_enter_start(kvm);
+		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+					  &entry, &level_state);
+		tdx_no_vcpus_enter_stop(kvm);
+	}
+
+	if (KVM_BUG_ON(err, kvm)) {
+		pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
+		return -EIO;
+	}
+	return 0;
+}
+
+static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,
+				      void *private_spt)
+{
+	struct page *page = virt_to_page(private_spt);
+	int ret;
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1953

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+
+	if (KVM_BUG_ON(to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE ||
+		       level != PG_LEVEL_2M, kvm))
+		return -EINVAL;
+
+	ret = tdx_sept_zap_private_spte(kvm, gfn, level, page);

I don't quite follow why you pass 'private_spt' to tdx_sept_zap_private_spte(),

Simply because tdx_sept_zap_private_spte() requires a "page", which is actually not used by tdx_sept_zap_private_spte() in the split path.

Quoted 2 lines

but it doesn't matter anymore since it's gone
in Sean's latest tree.

Right.

author
#50 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 9 lines · Binbin Wu
Quoted 3 lines · Yan Zhao
Quoted 2 lines · Kai
Quoted 1 line · Yan Zhao

Implement the split_external_spt hook to enable huge page splitting for

Nit:
split_external_spt(), similar as Kai mentioned in patch 9.

Ok. I'll refer it to kvm_x86_ops.split_external_spt().

#51 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

In reply to Yan Zhao
Quoted 40 lines · Yan Zhao

Implement the split_external_spt hook to enable huge page splitting for TDX when kvm->mmu_lock is held for writing.

Invoke tdh_mem_range_block(), tdh_mem_track(), kicking off vCPUs, tdh_mem_page_demote() in sequence. All operations are performed under kvm->mmu_lock held for writing, similar to those in page removal.

Even with kvm->mmu_lock held for writing, tdh_mem_page_demote() may still contend with tdh_vp_enter() and potentially with the guest's S-EPT entry operations. Therefore, kick off other vCPUs and prevent tdh_vp_enter() from being called on them to ensure success on the second attempt. Use KVM_BUG_ON() for any other unexpected errors.

---
RFC v2:
- Split out the code to handle the error TDX_INTERRUPTED_RESTARTABLE.
- Rebased to 6.16.0-rc6 (the way of defining TDX hook changes).

RFC v1:
- Split patch for exclusive mmu_lock only,
- Invoke tdx_sept_zap_private_spte() and tdx_track() for splitting.
- Handled busy error of tdh_mem_page_demote() by kicking off vCPUs.

---
 arch/x86/kvm/vmx/tdx.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
arch/x86/kvm/vmx/tdx.c +2 -0
index 376287a2ddf4..8a60ba5b6595 100644
@@ -1915,6 +1915,50 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,
        return 0;
 }
 
+static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
+                                       enum pg_level level, struct page *page)

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1929

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+{
+       int tdx_level = pg_level_to_tdx_sept_level(level);
+       struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
+       gpa_t gpa = gfn_to_gpa(gfn);
+       u64 err, entry, level_state;
+
+       err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+                                 &entry, &level_state);
+
+       if (unlikely(tdx_operand_busy(err))) {

I was trying to test this code locally (without the DPAMT patches and with DPAMT disabled) and saw that sometimes tdh_mem_page_demote returns TDX_INTERRUPTED_RESTARTABLE. Looking at the TDX module code (version 1.5.16 from [1]) I see that demote and promote are the only seamcalls that return TDX_INTERRUPTED_RESTARTABLE so it wasn't handled by KVM until now.

I added manual handling for it and it's working correctly. Note that my change is on top of a rebase to the latest version:

@@ -1989,9 +1989,16 @@ static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,

        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;

+ int i = 0;

- err = tdh_do_no_vcpus(tdh_mem_page_demote, kvm, &kvm_tdx->td, gpa,
+ while (i < TDX_SEAMCALL_RETRIES) {
+ err = tdh_do_no_vcpus(tdh_mem_page_demote, kvm,
&kvm_tdx->td, gpa,

                              tdx_level, page, &entry, &level_state);

+ if (err != TDX_INTERRUPTED_RESTARTABLE)
+ break;
+ i++;
+ }
+
if (TDX_BUG_ON_2(err, TDH_MEM_PAGE_DEMOTE, entry, level_state, kvm))
return -EIO;

[1] https://github.com/intel/confidential-computing.tdx.tdx-module

Quoted 46 lines
arch/x86/kvm/vmx/tdx.c +33 -1
+               tdx_no_vcpus_enter_start(kvm);
+               err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
+                                         &entry, &level_state);
+               tdx_no_vcpus_enter_stop(kvm);
+       }
+
+       if (KVM_BUG_ON(err, kvm)) {
+               pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
+               return -EIO;
+       }
+       return 0;
+}
+
+static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,
+                                     void *private_spt)
+{
+       struct page *page = virt_to_page(private_spt);
+       int ret;
+
+       if (KVM_BUG_ON(to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE ||
+                      level != PG_LEVEL_2M, kvm))
+               return -EINVAL;
+
+       ret = tdx_sept_zap_private_spte(kvm, gfn, level, page);
+       if (ret <= 0)
+               return ret;
+
+       tdx_track(kvm);
+
+       return tdx_spte_demote_private_spte(kvm, gfn, level, page);
+}
+
 static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
                                        enum pg_level level, kvm_pfn_t pfn)
 {
@@ -3668,5 +3712,6 @@ void __init tdx_hardware_setup(void)
        vt_x86_ops.set_external_spte = tdx_sept_set_private_spte;
        vt_x86_ops.free_external_spt = tdx_sept_free_private_spt;
        vt_x86_ops.remove_external_spte = tdx_sept_remove_private_spte;
+       vt_x86_ops.split_external_spt = tdx_sept_split_private_spt;
        vt_x86_ops.protected_apic_has_interrupt = tdx_protected_apic_has_interrupt;
 }
--

2.43.2

#52 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 6 lines · Sagi Shahar

I was trying to test this code locally (without the DPAMT patches and with DPAMT disabled) and saw that sometimes tdh_mem_page_demote returns TDX_INTERRUPTED_RESTARTABLE. Looking at the TDX module code (version 1.5.16 from [1]) I see that demote and promote are the only seamcalls that return TDX_INTERRUPTED_RESTARTABLE so it wasn't handled by KVM until now.

Did you see "Open 3" in the coverletter?

#53 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 10 lines · Rick P
Quoted 6 lines · Sagi Shahar

I was trying to test this code locally (without the DPAMT patches and with DPAMT disabled) and saw that sometimes tdh_mem_page_demote returns TDX_INTERRUPTED_RESTARTABLE. Looking at the TDX module code (version 1.5.16 from [1]) I see that demote and promote are the only seamcalls that return TDX_INTERRUPTED_RESTARTABLE so it wasn't handled by KVM until now.

Did you see "Open 3" in the coverletter?

I tested the code using TDX module 1.5.24 which is the latest one we got. Is there a newer TDX module that supports this new functionality?

author
#54 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 11 lines · Sagi Shahar
Quoted 9 lines · Rick P
Quoted 6 lines · Sagi Shahar

I was trying to test this code locally (without the DPAMT patches and with DPAMT disabled) and saw that sometimes tdh_mem_page_demote returns TDX_INTERRUPTED_RESTARTABLE. Looking at the TDX module code (version 1.5.16 from [1]) I see that demote and promote are the only seamcalls that return TDX_INTERRUPTED_RESTARTABLE so it wasn't handled by KVM until now.

Quoted 4 lines
Quoted 1 line

Did you see "Open 3" in the coverletter?

I tested the code using TDX module 1.5.24 which is the latest one we got. Is there a newer TDX module that supports this new functionality?

AFAIK, TDX module 1.5.28 is the earliest version that enumerates TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY (bit 51) and disables TDX_INTERRUPTED_RESTARTABLE when there're no L2 TDs. (Please check the discussions at [1]).

Looks 1.5.28.04 was just released (internally?), with release note saying "Ensure TDH.MEM.PAGE.DEMOTE forward progress for non partitioned TDs".

Not sure if you can check it.

[1] https://lore.kernel.org/all/[email protected]/

#55 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 17 lines · Yan Zhao
Quoted 11 lines · Sagi Shahar
Quoted 9 lines · Rick P
Quoted 6 lines · Sagi Shahar

I was trying to test this code locally (without the DPAMT patches and with DPAMT disabled) and saw that sometimes tdh_mem_page_demote returns TDX_INTERRUPTED_RESTARTABLE. Looking at the TDX module code (version 1.5.16 from [1]) I see that demote and promote are the only seamcalls that return TDX_INTERRUPTED_RESTARTABLE so it wasn't handled by KVM until now.

Quoted 4 lines
Quoted 1 line

Did you see "Open 3" in the coverletter?

I tested the code using TDX module 1.5.24 which is the latest one we got. Is there a newer TDX module that supports this new functionality?

Quoted 8 lines

AFAIK, TDX module 1.5.28 is the earliest version that enumerates TDX_FEATURES0.ENHANCED_DEMOTE_INTERRUPTIBILITY (bit 51) and disables TDX_INTERRUPTED_RESTARTABLE when there're no L2 TDs. (Please check the discussions at [1]).

Looks 1.5.28.04 was just released (internally?), with release note saying "Ensure TDH.MEM.PAGE.DEMOTE forward progress for non partitioned TDs".

Thanks. I don't have access to the 1.5.28.04 module and we need the code to work with the 1.5.24 module as well based on our timeline so I guess we can just add the retries locally for now.

Do you see any issue with retrying the operation in case of TDX_INTERRUPTED_RESTARTABLE? From what I saw this is not just a theoretical race but happens every time I try to boot a VM, even for a small VM with 4 VCPUs and 8GB of memory.

Quoted 4 lines

Not sure if you can check it.

[1] https://lore.kernel.org/all/[email protected]/

#56 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 7 lines · Sagi Shahar

Thanks. I don't have access to the 1.5.28.04 module and we need the code to work with the 1.5.24 module as well based on our timeline so I guess we can just add the retries locally for now.

Do you see any issue with retrying the operation in case of TDX_INTERRUPTED_RESTARTABLE?

Yan has been testing with a similar workaround. See "[DROP ME] x86/virt/tdx:
Loop for TDX_INTERRUPTED_RESTARTABLE in tdh_mem_page_demote()".

With TDX_INTERRUPTED_RESTARTABLE compared to RESUMABLE, the problem is that there is no guarantee it will make forward progress. So looping during an interrupt storm would halt the process context in an unusual way.

So the two kernel side options we discussed were loop forever, or loop for a certain amount of times and KVM_BUG_ON()/warn (like you had). They have different problems - unbounded loop vs potentially killing the TD for unrelated host behavior. So that is how we came to the decision to rely on TDX module changes for the long term upstream solution.

You could also see this thread that touches on disabling interrupts around the seamcall:
https://lore.kernel.org/kvm/[email protected]/

However it does not help the NMI case. Do you know which you are hitting?

Quoted 3 lines

From what I saw this is not just a
theoretical race but happens every time I try to boot a VM

Oh, interesting.

Quoted 2 lines

, even for a
small VM with 4 VCPUs and 8GB of memory.

It probably more matters what else is happening on the system to cause a host interrupt.

author
#57 lore raw

Re: [RFC PATCH v2 10/23] KVM: TDX: Enable huge page splitting under write kvm->mmu_lock

Quoted 3 lines · Rick P
Quoted 2 lines · Sagi Shahar

From what I saw this is not just a
theoretical race but happens every time I try to boot a VM

I don't think we mentioned that the retry for TDX_INTERRUPTED_RESTARTABLE is theoretical, did we? :)

On my SPR, to boot a VM with 8 vCPUs 8 GB memory, on average there are 271.4 demotes, with 1.1 retry for TDX_INTERRUPTED_RESTARTABLE.

   #    | demote cnt | retry cnt (for TDX_INTERRUPTED_RESTARTABLE)
--------|------------|----------
   1    |    271     |    2
   2    |    273     |    0
   3    |    271     |    1
   4    |    270     |    2
   5    |    271     |    3
   6    |    271     |    0
   7    |    274     |    0
   8    |    270     |    0
   9    |    271     |    2
  10    |    272     |    0
Quoted 7 lines

Oh, interesting.

Quoted 2 lines

, even for a
small VM with 4 VCPUs and 8GB of memory.

It probably more matters what else is happening on the system to cause a host interrupt.

Agreed.

author
#58 lore raw

RFC PATCH v2 11/23 KVM: x86: Reject splitting huge pages under shared mmu_lock for mirror root

In reply to Yan Zhao

While removing the KVM_BUG_ON() for the mirror root before invoking tdp_mmu_split_huge_page() in the fault path, update the hook split_external_spt to pass in shared mmu_lock info and invoke the hook in set_external_spte_present() on splitting is detected. Reject the splitting in TDX if the splitting is under shared mmu_lock.

TDX requires different handling for splitting under shared or exclusive mmu_lock.

Under a shared mmu_lock, TDX cannot kick off all vCPUs to avoid BUSY error from tdh_mem_page_demote(). As the current TDX module requires tdh_mem_range_block() to be invoked before each tdh_mem_page_demote(), if a BUSY error occurs, TDX must call tdh_mem_range_unblock() before returning the error to the KVM MMU core to roll back the old SPTE and retry. However, tdh_mem_range_unblock() may also fail due to contention.

Reject splitting huge pages under shared mmu_lock for mirror root in TDX rather than KVM_BUG_ON() in KVM MMU core to allow for future real implementation of demote under shared mmu_lock once non-blocking demote is available.

---
RFC v2:
- WARN_ON_ONCE() and return error in tdx_sept_split_private_spt() if it's invoked under shared mmu_lock. (rather than increase the next fault's max_level in current vCPU via tdx->violation_gfn_start/end and tdx->violation_request_level).
- TODO: Perform the real implementation of demote under shared mmu_lock when new version of TDX module supporting non-blocking demote is available.

RFC v1:
- New patch.

---
 arch/x86/include/asm/kvm_host.h |  2 +-
 arch/x86/kvm/mmu/tdp_mmu.c      | 45 ++++++++++++++++++++-------------
 arch/x86/kvm/vmx/tdx.c          |  8 +++++-
 3 files changed, 36 insertions(+), 19 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -1
index e431ce0e3180..6cb5b422dd1d 100644
@@ -1841,7 +1841,7 @@ struct kvm_x86_ops {
 
 	/* Split the external page table into smaller page tables */
 	int (*split_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
-				  void *external_spt);
+				  void *external_spt, bool mmu_lock_shared);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/tdp_mmu.c +28 -17

87 lines not on this page · raw message

arch/x86/kvm/vmx/tdx.c +7 -1
index 8a60ba5b6595..035d81275be4 100644
@@ -1941,7 +1941,7 @@ static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
 }
 
 static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,
-				      void *private_spt)
+				      void *private_spt, bool mmu_lock_shared)
 {
 	struct page *page = virt_to_page(private_spt);
 	int ret;
@@ -1950,6 +1950,12 @@ static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level
 		       level != PG_LEVEL_2M, kvm))
 		return -EINVAL;
 
+	if (WARN_ON_ONCE(mmu_lock_shared)) {
+		pr_warn_once("Splitting of GFN %llx level %d under shared lock occurs when KVM does not support it yet\n",
+			     gfn, level);
+		return -EOPNOTSUPP;
+	}
+
 	ret = tdx_sept_zap_private_spte(kvm, gfn, level, page);
 	if (ret <= 0)
 		return ret;
Signature · 1 line
2.43.2
#59 lore raw

Re: [RFC PATCH v2 11/23] KVM: x86: Reject splitting huge pages under shared mmu_lock for mirror root

Quoted 16 lines · Yan Zhao

While removing the KVM_BUG_ON() for the mirror root before invoking tdp_mmu_split_huge_page() in the fault path, update the hook split_external_spt to pass in shared mmu_lock info and invoke the hook in set_external_spte_present() on splitting is detected. Reject the splitting in TDX if the splitting is under shared mmu_lock.

TDX requires different handling for splitting under shared or exclusive mmu_lock.

Under a shared mmu_lock, TDX cannot kick off all vCPUs to avoid BUSY error from tdh_mem_page_demote(). As the current TDX module requires tdh_mem_range_block() to be invoked before each tdh_mem_page_demote(), if a BUSY error occurs, TDX must call tdh_mem_range_unblock() before returning the error to the KVM MMU core to roll back the old SPTE and retry. However, tdh_mem_range_unblock() may also fail due to contention.

Quoted 4 lines

Reject splitting huge pages under shared mmu_lock for mirror root in TDX rather than KVM_BUG_ON() in KVM MMU core to allow for future real implementation of demote under shared mmu_lock once non-blocking demote is available.

Prefer "blockless" used in the cover letter to non-blocking.

[...]

author
#60 lore raw

RFC PATCH v2 12/23 KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

In reply to Yan Zhao

Introduce kvm_split_cross_boundary_leafs() to split huge leaf entries that cross the boundary of a specified range.

Splitting huge leaf entries that cross the boundary is essential before zapping the range in the mirror root. This ensures that the subsequent zap operation does not affect any GFNs outside the specified range. This is crucial for the mirror root, as the private page table requires the guest's ACCEPT operation after a GFN faults back.

The core of kvm_split_cross_boundary_leafs() leverages the main logic from tdp_mmu_split_huge_pages_root(). It traverses the specified root and splits huge leaf entries if they cross the range boundary. When splitting is necessary, kvm->mmu_lock is temporarily released for memory allocation, which means returning -ENOMEM is possible.

---
RFC v2:
- Rename the API to kvm_split_cross_boundary_leafs().
- Make the API to be usable for direct roots or under shared mmu_lock.
- Leverage the main logic from tdp_mmu_split_huge_pages_root(). (Rick)

RFC v1:
- Split patch.
- introduced API kvm_split_boundary_leafs(), refined the logic and simplified the code.

---
 arch/x86/kvm/mmu/mmu.c     | 27 +++++++++++++++
 arch/x86/kvm/mmu/tdp_mmu.c | 68 ++++++++++++++++++++++++++++++++++++--
 arch/x86/kvm/mmu/tdp_mmu.h |  3 ++
 include/linux/kvm_host.h   |  2 ++
 4 files changed, 97 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +27 -0
index 9182192daa3a..13910ae05f76 100644
@@ -1647,6 +1647,33 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm,
 				 start, end - 1, can_yield, true, flush);
 }
 
+/*
+ * Split large leafs crossing the boundary of the specified range
+ *
+ * Return value:
+ * 0 : success, no flush is required;
+ * 1 : success, flush is required;
+ * <0: failure.
+ */
+int kvm_split_cross_boundary_leafs(struct kvm *kvm, struct kvm_gfn_range *range,
+				   bool shared)
+{
+	bool ret = 0;
+
+	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
+			    lockdep_is_held(&kvm->slots_lock) ||
+			    srcu_read_lock_held(&kvm->srcu));
+
+	if (!range->may_block)
+		return -EOPNOTSUPP;
+
+	if (tdp_mmu_enabled)
+		ret = kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(kvm, range, shared);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_split_cross_boundary_leafs);
+
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool flush = false;
arch/x86/kvm/mmu/tdp_mmu.c +65 -3

115 lines not on this page · raw message

arch/x86/kvm/mmu/tdp_mmu.h +3 -0
index 52acf99d40a0..332d47cce714 100644
@@ -69,6 +69,9 @@ void kvm_tdp_mmu_zap_all(struct kvm *kvm);
 void kvm_tdp_mmu_invalidate_roots(struct kvm *kvm,
 				  enum kvm_tdp_mmu_root_types root_types);
 void kvm_tdp_mmu_zap_invalidated_roots(struct kvm *kvm, bool shared);
+int kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(struct kvm *kvm,
+						     struct kvm_gfn_range *range,
+						     bool shared);
 
 int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault);
 
include/linux/kvm_host.h +2 -0
index fb79d2b7decd..6137b76341e1 100644
@@ -273,6 +273,8 @@ struct kvm_gfn_range {
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
+int kvm_split_cross_boundary_leafs(struct kvm *kvm, struct kvm_gfn_range *range,
+				   bool shared);
 #endif
 
 enum {
Signature · 1 line
2.43.2
#61 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 83 lines · Yan Zhao

Introduce kvm_split_cross_boundary_leafs() to split huge leaf entries that cross the boundary of a specified range.

Splitting huge leaf entries that cross the boundary is essential before zapping the range in the mirror root. This ensures that the subsequent zap operation does not affect any GFNs outside the specified range. This is crucial for the mirror root, as the private page table requires the guest's ACCEPT operation after a GFN faults back.

The core of kvm_split_cross_boundary_leafs() leverages the main logic from tdp_mmu_split_huge_pages_root(). It traverses the specified root and splits huge leaf entries if they cross the range boundary. When splitting is necessary, kvm->mmu_lock is temporarily released for memory allocation, which means returning -ENOMEM is possible.

---
RFC v2:
- Rename the API to kvm_split_cross_boundary_leafs().
- Make the API to be usable for direct roots or under shared mmu_lock.
- Leverage the main logic from tdp_mmu_split_huge_pages_root(). (Rick)

RFC v1:
- Split patch.
- introduced API kvm_split_boundary_leafs(), refined the logic and simplified the code.

---
  arch/x86/kvm/mmu/mmu.c     | 27 +++++++++++++++
  arch/x86/kvm/mmu/tdp_mmu.c | 68 ++++++++++++++++++++++++++++++++++++--
  arch/x86/kvm/mmu/tdp_mmu.h |  3 ++
  include/linux/kvm_host.h   |  2 ++
  4 files changed, 97 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +27 -0
index 9182192daa3a..13910ae05f76 100644
@@ -1647,6 +1647,33 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm,
  				 start, end - 1, can_yield, true, flush);
  }
 
+/*
+ * Split large leafs crossing the boundary of the specified range
+ *
+ * Return value:
+ * 0 : success, no flush is required;
+ * 1 : success, flush is required;
+ * <0: failure.
+ */
+int kvm_split_cross_boundary_leafs(struct kvm *kvm, struct kvm_gfn_range *range,
+				   bool shared)
+{
+	bool ret = 0;
+
+	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
+			    lockdep_is_held(&kvm->slots_lock) ||
+			    srcu_read_lock_held(&kvm->srcu));
+
+	if (!range->may_block)
+		return -EOPNOTSUPP;
+
+	if (tdp_mmu_enabled)
+		ret = kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(kvm, range, shared);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_split_cross_boundary_leafs);
+
  bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
  {
  	bool flush = false;
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
index ce49cc850ed5..62a09a9655c3 100644
@@ -1574,10 +1574,17 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter,
  	return ret;
  }
 
+static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end)
+{

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, · line 1587

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -1
+	return !(iter->gfn >= start &&
+		 (iter->gfn + KVM_PAGES_PER_HPAGE(iter->level)) <= end);
+}
+
  static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  					 struct kvm_mmu_page *root,
  					 gfn_t start, gfn_t end,
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

s/only_cross_bounday/only_cross_boundary

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1604

Quoted 13 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -0
  {
  	struct kvm_mmu_page *sp = NULL;
  	struct tdp_iter iter;
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.

I think API wise, it's a bit confusing, although it's a local API. If just look at the API without digging into the function implementation, my initial thought is *flush will tell whether TLB flush is needed or not.

Just update *flush unconditionally? Or move the comment as the description for the function to call it out?

I have thought another option to combine the two inputs, i.e., if *flush is a valid pointer, it means it's for only_cross_boundary. Otherwise, just passing NULL. But then I felt it was a bit risky to reply on the pointer to indicate the scenario.

Quoted 34 lines
arch/x86/kvm/mmu/tdp_mmu.c +11 -1
+	 *
  	 * Since the TDP iterator uses a pre-order traversal, we are guaranteed
  	 * to visit an SPTE before ever visiting its children, which means we
  	 * will correctly recursively split huge pages that are more than one
@@ -1597,12 +1611,19 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 */
  	for_each_tdp_pte_min_level(iter, kvm, root, target_level + 1, start, end) {
  retry:
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, false, shared))
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, *flush, shared)) {
+			if (only_cross_bounday)
+				*flush = false;
  			continue;
+		}
 
  		if (!is_shadow_present_pte(iter.old_spte) || !is_large_pte(iter.old_spte))
  			continue;
 
+		if (only_cross_bounday &&
+		    !iter_cross_boundary(&iter, start, end))
+			continue;
+
  		if (!sp) {
  			rcu_read_unlock();
 
@@ -1637,6 +1658,8 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  			goto retry;
 
  		sp = NULL;
+		if (only_cross_bounday)
+			*flush = true;
  	}
 
  	rcu_read_unlock();

[...]

author
#62 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 87 lines · Binbin Wu
Quoted 84 lines · Yan Zhao

Introduce kvm_split_cross_boundary_leafs() to split huge leaf entries that cross the boundary of a specified range.

Splitting huge leaf entries that cross the boundary is essential before zapping the range in the mirror root. This ensures that the subsequent zap operation does not affect any GFNs outside the specified range. This is crucial for the mirror root, as the private page table requires the guest's ACCEPT operation after a GFN faults back.

The core of kvm_split_cross_boundary_leafs() leverages the main logic from tdp_mmu_split_huge_pages_root(). It traverses the specified root and splits huge leaf entries if they cross the range boundary. When splitting is necessary, kvm->mmu_lock is temporarily released for memory allocation, which means returning -ENOMEM is possible.

---
RFC v2:
- Rename the API to kvm_split_cross_boundary_leafs().
- Make the API to be usable for direct roots or under shared mmu_lock.
- Leverage the main logic from tdp_mmu_split_huge_pages_root(). (Rick)

RFC v1:
- Split patch.
- introduced API kvm_split_boundary_leafs(), refined the logic and simplified the code.

---
  arch/x86/kvm/mmu/mmu.c     | 27 +++++++++++++++
  arch/x86/kvm/mmu/tdp_mmu.c | 68 ++++++++++++++++++++++++++++++++++++--
  arch/x86/kvm/mmu/tdp_mmu.h |  3 ++
  include/linux/kvm_host.h   |  2 ++
  4 files changed, 97 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu/mmu.c +27 -0
index 9182192daa3a..13910ae05f76 100644
@@ -1647,6 +1647,33 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm,
  				 start, end - 1, can_yield, true, flush);
  }
+/*
+ * Split large leafs crossing the boundary of the specified range
+ *
+ * Return value:
+ * 0 : success, no flush is required;
+ * 1 : success, flush is required;
+ * <0: failure.
+ */
+int kvm_split_cross_boundary_leafs(struct kvm *kvm, struct kvm_gfn_range *range,
+				   bool shared)
+{
+	bool ret = 0;
+
+	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
+			    lockdep_is_held(&kvm->slots_lock) ||
+			    srcu_read_lock_held(&kvm->srcu));
+
+	if (!range->may_block)
+		return -EOPNOTSUPP;
+
+	if (tdp_mmu_enabled)
+		ret = kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(kvm, range, shared);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_split_cross_boundary_leafs);
+
  bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
  {
  	bool flush = false;
arch/x86/kvm/mmu/tdp_mmu.c +5 -0
index ce49cc850ed5..62a09a9655c3 100644
@@ -1574,10 +1574,17 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter,
  	return ret;
  }
+static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end)
+{
+	return !(iter->gfn >= start &&
+		 (iter->gfn + KVM_PAGES_PER_HPAGE(iter->level)) <= end);
+}
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter, · line 1587

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +3 -1
+
  static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  					 struct kvm_mmu_page *root,
  					 gfn_t start, gfn_t end,
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

s/only_cross_bounday/only_cross_boundary

Will fix.

Quoted 18 lines

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1604

Quoted 13 lines
arch/x86/kvm/mmu/tdp_mmu.c +6 -0
  {
  	struct kvm_mmu_page *sp = NULL;
  	struct tdp_iter iter;
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.

I think API wise, it's a bit confusing, although it's a local API. If just look at the API without digging into the function implementation, my initial thought is *flush will tell whether TLB flush is needed or not.

Quoted 7 lines

Just update *flush unconditionally? Or move the comment as the description for the function to call it out?

I have thought another option to combine the two inputs, i.e., if *flush is a valid pointer, it means it's for only_cross_boundary. Otherwise, just passing NULL. But then I felt it was a bit risky to reply on the pointer to indicate the scenario.

I feel it's better not to combine flush and only_cross_boundary. Will add a function description to tdp_mmu_split_huge_pages_root().

Quoted 30 lines
Quoted 29 lines
arch/x86/kvm/mmu/tdp_mmu.c +11 -1
+	 *
  	 * Since the TDP iterator uses a pre-order traversal, we are guaranteed
  	 * to visit an SPTE before ever visiting its children, which means we
  	 * will correctly recursively split huge pages that are more than one
@@ -1597,12 +1611,19 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 */
  	for_each_tdp_pte_min_level(iter, kvm, root, target_level + 1, start, end) {
  retry:
-		if (tdp_mmu_iter_cond_resched(kvm, &iter, false, shared))
+		if (tdp_mmu_iter_cond_resched(kvm, &iter, *flush, shared)) {
+			if (only_cross_bounday)
+				*flush = false;
  			continue;
+		}
  		if (!is_shadow_present_pte(iter.old_spte) || !is_large_pte(iter.old_spte))
  			continue;
+		if (only_cross_bounday &&
+		    !iter_cross_boundary(&iter, start, end))
+			continue;
+
  		if (!sp) {
  			rcu_read_unlock();
@@ -1637,6 +1658,8 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  			goto retry;
  		sp = NULL;
+		if (only_cross_bounday)
+			*flush = true;
  	}
  	rcu_read_unlock();

[...]

#63 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

In reply to Yan Zhao
Quoted 10 lines · Yan Zhao

 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end,

arch/x86/kvm/mmu/tdp_mmu.c +2 -1
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

 {
  struct kvm_mmu_page *sp = NULL;
  struct tdp_iter iter;

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1605

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +7 -0
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.
+	 *

s/only_cross_bounday/only_cross_boundary

From tdp_mmu_split_huge_pages_root()'s perspective, it's quite odd to only update 'flush' when 'only_cross_bounday' is true, because 'only_cross_bounday' can only results in less splitting.

I understand this is because splitting S-EPT mapping needs flush (at least before non-block DEMOTE is implemented?). Would it better to also let the caller to decide whether TLB flush is needed? E.g., we can make tdp_mmu_split_huge_pages_root() return whether any split has been done or not. I think this should also work?

author
#64 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 24 lines · Kai
Quoted 10 lines · Yan Zhao

 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end,

(quoted diff) +2 -1
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

 {
  struct kvm_mmu_page *sp = NULL;
  struct tdp_iter iter;

Review static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1605

Quoted 10 lines
(quoted diff) +7 -0
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.
+	 *

s/only_cross_bounday/only_cross_boundary

Quoted 3 lines

From tdp_mmu_split_huge_pages_root()'s perspective, it's quite odd to only update 'flush' when 'only_cross_bounday' is true, because 'only_cross_bounday' can only results in less splitting.

I have to say it's a reasonable point.

Quoted 2 lines

I understand this is because splitting S-EPT mapping needs flush (at least before non-block DEMOTE is implemented?). Would it better to also let the

Actually the flush is only required for !TDX cases.

For TDX, either the flush has been performed internally within tdx_sept_split_private_spt() or the flush is not required for future non-block DEMOTE. So, the flush in KVM core on the mirror root may be skipped as a future optimization for TDX if necessary.

Quoted 3 lines

caller to decide whether TLB flush is needed? E.g., we can make tdp_mmu_split_huge_pages_root() return whether any split has been done or not. I think this should also work?

Do you mean just skipping the changes in the below "Hunk 1"?

Since tdp_mmu_split_huge_pages_root() originally did not do flush by itself, which relied on the end callers (i.e.,kvm_mmu_slot_apply_flags(), kvm_clear_dirty_log_protect(), and kvm_get_dirty_log_protect()) to do the flush unconditionally, tdp_mmu_split_huge_pages_root() previously did not return whether any split has been done or not.

So, if we want callers of kvm_split_cross_boundary_leafs() to do flush only after splitting occurs, we have to return whether flush is required.

Then, in this patch, seems only the changes in "Hunk 1" can be dropped.

Hunk 1
-------------------------------

        for_each_tdp_pte_min_level(iter, kvm, root, target_level + 1, start, end) {

retry:
- if (tdp_mmu_iter_cond_resched(kvm, &iter, false, shared))
+ if (tdp_mmu_iter_cond_resched(kvm, &iter, *flush, shared)) {
+ if (only_cross_bounday)

+                               *flush = false;
                        continue;

+ }

                if (!is_shadow_present_pte(iter.old_spte) || !is_large_pte(iter.old_spte))
                        continue;

Hunk 2
-------------------------------
+ if (only_cross_bounday &&
+ !iter_cross_boundary(&iter, start, end))
+ continue;
+

                if (!sp) {
                        rcu_read_unlock();

Hunk 3
-------------------------------
@@ -1637,6 +1658,8 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,

                        goto retry;

                sp = NULL;
+               if (only_cross_bounday)
+                       *flush = true;
        }

Do you think dropping of "Hunk 1" is worthwhile? Would it be less odd if I make the following change?

arch/x86/kvm/mmu/tdp_mmu.c +3 -2
index 9f479832a981..7bc1d1a5f3ce 100644
@@ -1589,6 +1589,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
 {
        struct kvm_mmu_page *sp = NULL;
        struct tdp_iter iter;
+       bool caller_unconditional_flush = !only_cross_bounday;
 
        rcu_read_lock();
 
@@ -1613,7 +1614,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
        for_each_tdp_pte_min_level(iter, kvm, root, target_level + 1, start, end) {
 retry:
                if (tdp_mmu_iter_cond_resched(kvm, &iter, *flush, shared)) {
-                       if (only_cross_bounday)
+                       if (!caller_unconditional_flush)
                                *flush = false;
                        continue;
                }
@@ -1659,7 +1660,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                        goto retry;
 
                sp = NULL;
-               if (only_cross_bounday)
+               if (!caller_unconditional_flush)
                        *flush = true;
        }
 
#65 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 28 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 10 lines · Yan Zhao

 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end,

(quoted diff) +2 -1
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

 {
  struct kvm_mmu_page *sp = NULL;
  struct tdp_iter iter;

Review static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1605

Quoted 10 lines
(quoted diff) +7 -0
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.
+	 *

s/only_cross_bounday/only_cross_boundary

Quoted 3 lines

From tdp_mmu_split_huge_pages_root()'s perspective, it's quite odd to only update 'flush' when 'only_cross_bounday' is true, because 'only_cross_bounday' can only results in less splitting.

Quoted 9 lines

I have to say it's a reasonable point.

Quoted 2 lines

I understand this is because splitting S-EPT mapping needs flush (at least before non-block DEMOTE is implemented?). Would it better to also let the

Actually the flush is only required for !TDX cases.

For TDX, either the flush has been performed internally within tdx_sept_split_private_spt()

AFAICT tdx_sept_split_private_spt() only does tdh_mem_track(), so KVM should still kick all vCPUs out of guest mode so other vCPUs can actually flush the TLB?

Quoted 7 lines

or the flush is not required for future non-block
DEMOTE. So, the flush in KVM core on the mirror root may be skipped as a future optimization for TDX if necessary.

Quoted 3 lines

caller to decide whether TLB flush is needed? E.g., we can make tdp_mmu_split_huge_pages_root() return whether any split has been done or not. I think this should also work?

Quoted 7 lines

Do you mean just skipping the changes in the below "Hunk 1"?

Since tdp_mmu_split_huge_pages_root() originally did not do flush by itself, which relied on the end callers (i.e.,kvm_mmu_slot_apply_flags(), kvm_clear_dirty_log_protect(), and kvm_get_dirty_log_protect()) to do the flush unconditionally, tdp_mmu_split_huge_pages_root() previously did not return whether any split has been done or not.

Right. But making it return any split has been done doesn't harm.

Quoted 3 lines

So, if we want callers of kvm_split_cross_boundary_leafs() to do flush only after splitting occurs, we have to return whether flush is required.

But assuming we always return whether "split has been done", the caller can also effectively know whether the flush is needed.

Quoted 2 lines

Then, in this patch, seems only the changes in "Hunk 1" can be dropped.

I am thinking dropping both "Hunk 1" and "Hunk 3". This at least makes kvm_split_cross_boundary_leafs() more reasonable, IMHO.

Something like below:

@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct tdp_iter *iter,
static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
struct kvm_mmu_page *root, gfn_t start, gfn_t end,

-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
 {
        struct kvm_mmu_page *sp = NULL;
        struct tdp_iter iter;
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                if (!is_shadow_present_pte(iter.old_spte) ||
!is_large_pte(iter.old_spte))
                        continue;

+ if (only_cross_boundary && !iter_cross_boundary(&iter, start,
end))
+ continue;
+

                if (!sp) {
                        rcu_read_unlock();

@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,

                        goto retry;
 
                sp = NULL;
+               *split = true;
        }
 
        rcu_read_unlock();

Btw, I have to follow up this next week, since tomorrow is public holiday.

#66 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 31 lines · Kai

I am thinking dropping both "Hunk 1" and "Hunk 3".  This at least makes kvm_split_cross_boundary_leafs() more reasonable, IMHO.

Something like below:

(quoted diff) +3 -1
@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct
 tdp_iter *iter,
  static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                          struct kvm_mmu_page *root,
                                          gfn_t start, gfn_t end,
-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
  {
         struct kvm_mmu_page *sp = NULL;

        struct tdp_iter iter;

(quoted diff) +0 -0
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,

                if (!is_shadow_present_pte(iter.old_spte) || !is_large_pte(iter.old_spte))
                        continue;

+               if (only_cross_boundary && !iter_cross_boundary(&iter, start, end))
+                       continue;
+
                if (!sp) {
                        rcu_read_unlock();

arch/x86/kvm/mmu/tdp_mmu.c +0 -0
@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                         goto retry;
Quoted 4 lines


                sp = NULL;
+               *split = true;
        }

Forgot to say, if needed, we can update @split only when it is a valid pointer:

		if (split)
			*split = true;

This allows the caller to be able to just pass NULL when it doesn't care about whether split has been done.

author
#67 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

In reply to Huang, Kai
Quoted 35 lines · Kai
Quoted 34 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 10 lines · Yan Zhao

 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end,

(quoted diff) +2 -1
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

 {
  struct kvm_mmu_page *sp = NULL;
  struct tdp_iter iter;

Review static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1605

Quoted 10 lines
(quoted diff) +7 -0
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.
+	 *

s/only_cross_bounday/only_cross_boundary

Quoted 3 lines

From tdp_mmu_split_huge_pages_root()'s perspective, it's quite odd to only update 'flush' when 'only_cross_bounday' is true, because 'only_cross_bounday' can only results in less splitting.

I have to say it's a reasonable point.

Quoted 2 lines

I understand this is because splitting S-EPT mapping needs flush (at least before non-block DEMOTE is implemented?). Would it better to also let the

Actually the flush is only required for !TDX cases.

Quoted 7 lines
Quoted 3 lines

For TDX, either the flush has been performed internally within tdx_sept_split_private_spt()

AFAICT tdx_sept_split_private_spt() only does tdh_mem_track(), so KVM should still kick all vCPUs out of guest mode so other vCPUs can actually flush the TLB?

tdx_sept_split_private_spt() actually invokes tdx_track(), which performs the kicking off all vCPUs by invoking
"kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE)".

Quoted 57 lines
Quoted 7 lines

or the flush is not required for future non-block
DEMOTE. So, the flush in KVM core on the mirror root may be skipped as a future optimization for TDX if necessary.

Quoted 3 lines

caller to decide whether TLB flush is needed? E.g., we can make tdp_mmu_split_huge_pages_root() return whether any split has been done or not. I think this should also work?

Quoted 7 lines

Do you mean just skipping the changes in the below "Hunk 1"?

Since tdp_mmu_split_huge_pages_root() originally did not do flush by itself, which relied on the end callers (i.e.,kvm_mmu_slot_apply_flags(), kvm_clear_dirty_log_protect(), and kvm_get_dirty_log_protect()) to do the flush unconditionally, tdp_mmu_split_huge_pages_root() previously did not return whether any split has been done or not.

Right. But making it return any split has been done doesn't harm.

Quoted 3 lines

So, if we want callers of kvm_split_cross_boundary_leafs() to do flush only after splitting occurs, we have to return whether flush is required.

But assuming we always return whether "split has been done", the caller can also effectively know whether the flush is needed.

Quoted 2 lines

Then, in this patch, seems only the changes in "Hunk 1" can be dropped.

I am thinking dropping both "Hunk 1" and "Hunk 3". This at least makes kvm_split_cross_boundary_leafs() more reasonable, IMHO.

Something like below:

(quoted diff) +3 -1
@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct
 tdp_iter *iter,
 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                         struct kvm_mmu_page *root,
                                         gfn_t start, gfn_t end,
-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
 {
        struct kvm_mmu_page *sp = NULL;
        struct tdp_iter iter;
(quoted diff) +3 -0
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                if (!is_shadow_present_pte(iter.old_spte) ||
 !is_large_pte(iter.old_spte))
                        continue;
 
+               if (only_cross_boundary && !iter_cross_boundary(&iter, start,
 end))
+                       continue;
+
                if (!sp) {
                        rcu_read_unlock();

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1629

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                        goto retry;
 
                sp = NULL;
+               *split = true;
        }
 
        rcu_read_unlock();

This looks more reasonable for tdp_mmu_split_huge_pages_root();

Given that splitting only adds a new page to the paging structure (unlike page merging), I currently can't think of any current use cases that would be broken by the lack of TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock.

This is because:
1) if the split is triggered in a fault path, the hardware shouldn't have cached the old huge translation.
2) if the split is triggered in a zap or convert path,
- there shouldn't be concurrent faults on the range due to the protection of mmu_invalidate_range*.
- for concurrent splits on the same range, though the other vCPUs may temporally see stale huge TLB entries after they believe they have performed a split, they will be kicked off to flush the cache soon after tdp_mmu_split_huge_pages_root() returns in the first vCPU/host thread. This should be acceptable since I don't see any special guest needs that rely on pure splits.

So I tend to agree with your suggestion though the implementation in this patch is safer.

Quoted 1 line

Btw, I have to follow up this next week, since tomorrow is public holiday.

NP.

#68 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 40 lines · Yan Zhao
Quoted 39 lines · Kai
Quoted 37 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 10 lines · Yan Zhao

 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end,

(quoted diff) +2 -1
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

 {
  struct kvm_mmu_page *sp = NULL;
  struct tdp_iter iter;

Review static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1605

Quoted 10 lines
(quoted diff) +7 -0
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.
+	 *

s/only_cross_bounday/only_cross_boundary

Quoted 3 lines

From tdp_mmu_split_huge_pages_root()'s perspective, it's quite odd to only update 'flush' when 'only_cross_bounday' is true, because 'only_cross_bounday' can only results in less splitting.

I have to say it's a reasonable point.

Quoted 2 lines

I understand this is because splitting S-EPT mapping needs flush (at least before non-block DEMOTE is implemented?). Would it better to also let the

Actually the flush is only required for !TDX cases.

For TDX, either the flush has been performed internally within tdx_sept_split_private_spt()

Quoted 6 lines
Quoted 3 lines

AFAICT tdx_sept_split_private_spt() only does tdh_mem_track(), so KVM should still kick all vCPUs out of guest mode so other vCPUs can actually flush the TLB?

tdx_sept_split_private_spt() actually invokes tdx_track(), which performs the kicking off all vCPUs by invoking
"kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE)".

Oh thanks for the reminder.

Then I am kinda confused why do you need to return @flush, especially when 'only_cross_boundary' is true which is for TDX case?

So step back to where why this 'flush' is needed to be returned:

- For TDX ('only_cross_boundary == true'):

The caller doesn't need to flush TLB because it has already been done when huge page is actually split.

- For non-TDX case ('only_cross_boundary == false'):

AFAICT the only user of tdp_mmu_split_huge_pages_root() is "eager hugepage splitting" during log-dirty. And per per the current implementation there are two callers of tdp_mmu_split_huge_pages_root():

kvm_mmu_try_split_huge_pages()
kvm_mmu_slot_try_split_huge_pages()

But they are both void functions which neither return whether flush TLB is needed, nor do TLB flush internally.

So I am kinda confused.

Perhaps you mean for "shared memory of TDX guest", the caller will also pass 'only_cross_boundary == true' and the caller needs to perform TLB flush?

[...]

Quoted 42 lines
Quoted 28 lines

Something like below:

(quoted diff) +3 -1
@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct
 tdp_iter *iter,
 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                         struct kvm_mmu_page *root,
                                         gfn_t start, gfn_t end,
-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
 {
        struct kvm_mmu_page *sp = NULL;
        struct tdp_iter iter;
(quoted diff) +3 -0
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                if (!is_shadow_present_pte(iter.old_spte) ||
 !is_large_pte(iter.old_spte))
                        continue;
 
+               if (only_cross_boundary && !iter_cross_boundary(&iter, start,
 end))
+                       continue;
+
                if (!sp) {
                        rcu_read_unlock();

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1629

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                        goto retry;
 
                sp = NULL;
+               *split = true;
        }
 
        rcu_read_unlock();

This looks more reasonable for tdp_mmu_split_huge_pages_root();

Given that splitting only adds a new page to the paging structure (unlike page merging), I currently can't think of any current use cases that would be broken by the lack of TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock.

Quoted 12 lines

This is because:
1) if the split is triggered in a fault path, the hardware shouldn't have cached the old huge translation.
2) if the split is triggered in a zap or convert path,
- there shouldn't be concurrent faults on the range due to the protection of mmu_invalidate_range*.
- for concurrent splits on the same range, though the other vCPUs may temporally see stale huge TLB entries after they believe they have performed a split, they will be kicked off to flush the cache soon after tdp_mmu_split_huge_pages_root() returns in the first vCPU/host thread. This should be acceptable since I don't see any special guest needs that rely on pure splits.

Perhaps we should just go straight to the point:

What does "hugepage split" do, and what's the consequence of not flushing TLB.

Per make_small_spte(), the new child PTEs will carry all bits of hugepage PTE except they clear the 'hugepage bit (obviously)', and set the 'X' bit for NX hugepage thing.

That means if we leave the stale hugepage TLB, the CPU is still able to find the correct PFN and AFAICT there shouldn't be any other problem here. For any fault due to the stale hugepage TLB missing the 'X' permission, AFAICT KVM will just treat this as a spurious fault, which isn't nice but should have no harm.

Quoted 3 lines

So I tend to agree with your suggestion though the implementation in this patch is safer.

I am perhaps still missing something, as I am still trying to precisely understand in what cases you want to flush TLB when splitting hugepage.

I kinda tend to think you eventually want to flush TLB because eventually you want to _ZAP_. But needing to flush due to zap and needing to flush due to split is kinda different I think.

author
#69 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 65 lines · Kai
Quoted 40 lines · Yan Zhao
Quoted 39 lines · Kai
Quoted 37 lines · Yan Zhao
Quoted 24 lines · Kai
Quoted 10 lines · Yan Zhao

 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, struct kvm_mmu_page *root, gfn_t start, gfn_t end,

(quoted diff) +2 -1
-					 int target_level, bool shared)
+					 int target_level, bool shared,
+					 bool only_cross_bounday, bool *flush)

 {
  struct kvm_mmu_page *sp = NULL;
  struct tdp_iter iter;

Review static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1605

Quoted 10 lines
(quoted diff) +7 -0
@@ -1589,6 +1596,13 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  	 * level into one lower level. For example, if we encounter a 1GB page
  	 * we split it into 512 2MB pages.
  	 *
+	 * When only_cross_bounday is true, just split huge pages above the
+	 * target level into one lower level if the huge pages cross the start
+	 * or end boundary.
+	 *
+	 * No need to update @flush for !only_cross_bounday cases, which rely
+	 * on the callers to do the TLB flush in the end.
+	 *

s/only_cross_bounday/only_cross_boundary

Quoted 3 lines

From tdp_mmu_split_huge_pages_root()'s perspective, it's quite odd to only update 'flush' when 'only_cross_bounday' is true, because 'only_cross_bounday' can only results in less splitting.

I have to say it's a reasonable point.

Quoted 2 lines

I understand this is because splitting S-EPT mapping needs flush (at least before non-block DEMOTE is implemented?). Would it better to also let the

Actually the flush is only required for !TDX cases.

For TDX, either the flush has been performed internally within tdx_sept_split_private_spt()

Quoted 6 lines
Quoted 3 lines

AFAICT tdx_sept_split_private_spt() only does tdh_mem_track(), so KVM should still kick all vCPUs out of guest mode so other vCPUs can actually flush the TLB?

tdx_sept_split_private_spt() actually invokes tdx_track(), which performs the kicking off all vCPUs by invoking
"kvm_make_all_cpus_request(kvm, KVM_REQ_OUTSIDE_GUEST_MODE)".

Oh thanks for the reminder.

Then I am kinda confused why do you need to return @flush, especially when 'only_cross_boundary' is true which is for TDX case?
So step back to where why this 'flush' is needed to be returned:

- For TDX ('only_cross_boundary == true'):

The caller doesn't need to flush TLB because it has already been done when huge page is actually split.

- For non-TDX case ('only_cross_boundary == false'):

AFAICT the only user of tdp_mmu_split_huge_pages_root() is "eager hugepage splitting" during log-dirty. And per per the current implementation there are two callers of tdp_mmu_split_huge_pages_root():

Quoted 5 lines

kvm_mmu_try_split_huge_pages()
kvm_mmu_slot_try_split_huge_pages()

But they are both void functions which neither return whether flush TLB is needed, nor do TLB flush internally.

Actually callers of the two void functions do the TLB flush unconditionally in the end, i.e, in
kvm_mmu_slot_apply_flags(),
kvm_clear_dirty_log_protect(), and kvm_get_dirty_log_protect()).

Quoted 4 lines

So I am kinda confused.

Perhaps you mean for "shared memory of TDX guest", the caller will also pass 'only_cross_boundary == true' and the caller needs to perform TLB flush?

Sorry for the confusion.

Currently 'only_cross_boundary == true' is only for TDX private memory.

Returning flush is because kvm_split_cross_boundary_leafs() is potentially possible to be invoked for non-TDX cases as well in future (though currently it's only invoked for TDX alone). When that occurs, it's better to return flush to avoid the caller having to do flush unconditionally.

Another reason is to keep consistency with tdp_mmu_zap_leafs(), which returns flush without differentiate whether the zap is for a mirror root not not. So, though kvm_mmu_remote_flush() on mirror root is not necessary, it's intentionally left for future optimization.

Quoted 61 lines

[...]

Quoted 42 lines
Quoted 28 lines

Something like below:

(quoted diff) +3 -1
@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct
 tdp_iter *iter,
 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                         struct kvm_mmu_page *root,
                                         gfn_t start, gfn_t end,
-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
 {
        struct kvm_mmu_page *sp = NULL;
        struct tdp_iter iter;
(quoted diff) +3 -0
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                if (!is_shadow_present_pte(iter.old_spte) ||
 !is_large_pte(iter.old_spte))
                        continue;
 
+               if (only_cross_boundary && !iter_cross_boundary(&iter, start,
 end))
+                       continue;
+
                if (!sp) {
                        rcu_read_unlock();

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1629

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                        goto retry;
 
                sp = NULL;
+               *split = true;
        }
 
        rcu_read_unlock();

This looks more reasonable for tdp_mmu_split_huge_pages_root();

Given that splitting only adds a new page to the paging structure (unlike page merging), I currently can't think of any current use cases that would be broken by the lack of TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock.

Quoted 12 lines

This is because:
1) if the split is triggered in a fault path, the hardware shouldn't have cached the old huge translation.
2) if the split is triggered in a zap or convert path,
- there shouldn't be concurrent faults on the range due to the protection of mmu_invalidate_range*.
- for concurrent splits on the same range, though the other vCPUs may temporally see stale huge TLB entries after they believe they have performed a split, they will be kicked off to flush the cache soon after tdp_mmu_split_huge_pages_root() returns in the first vCPU/host thread. This should be acceptable since I don't see any special guest needs that rely on pure splits.

Perhaps we should just go straight to the point:

What does "hugepage split" do, and what's the consequence of not flushing TLB.

Quoted 8 lines

Per make_small_spte(), the new child PTEs will carry all bits of hugepage PTE except they clear the 'hugepage bit (obviously)', and set the 'X' bit for NX hugepage thing.

That means if we leave the stale hugepage TLB, the CPU is still able to find the correct PFN and AFAICT there shouldn't be any other problem here. For any fault due to the stale hugepage TLB missing the 'X' permission, AFAICT KVM will just treat this as a spurious fault, which isn't nice but should have no harm.

Right, that isn't nice, though no harm.

Besides, I'm thinking of a scenario which is not currently existing though.

CPU 0 CPU 1
a1. split pages
a2. write protect pages
b1. split pages b2. write protect pages b3. start dirty page tracking
a3. flush TLB
a4. start dirty page tracking

If CPU 1 does not flush TLB after b2 (e.g., due to it finds the pages have been split and write protected by a1&a2), it will miss some dirty pages.

Currently CPU 1 always flush TLB before b3 unconditionally, so there's no problem.

Quoted 9 lines
Quoted 2 lines

So I tend to agree with your suggestion though the implementation in this patch is safer.

I am perhaps still missing something, as I am still trying to precisely understand in what cases you want to flush TLB when splitting hugepage.

I kinda tend to think you eventually want to flush TLB because eventually you want to _ZAP_. But needing to flush due to zap and needing to flush due to split is kinda different I think.

Though I currently couldn't find any use cases that depend on split alone, e.g. if there's any feature requiring the pages must be 4KB without any additional permission changes, I just wanted to make the code safer in case I missed any edge cases.

We surely don't want the window for CPUs to see huge pages and small pages lasts long.

Flushing TLB before releasing the mmu_lock allows other threads operating on the same range to see updated translations timely.

#70 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

On 11/18/2025 2:30 PM, Yan Zhao wrote:
[...]

Quoted 55 lines
Quoted 55 lines
Quoted 41 lines
Quoted 27 lines

Something like below:

(quoted diff) +3 -1
@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct
 tdp_iter *iter,
  static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                          struct kvm_mmu_page *root,
                                          gfn_t start, gfn_t end,
-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
  {
         struct kvm_mmu_page *sp = NULL;
         struct tdp_iter iter;
(quoted diff) +3 -0
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                 if (!is_shadow_present_pte(iter.old_spte) ||
 !is_large_pte(iter.old_spte))
                         continue;
 
+               if (only_cross_boundary && !iter_cross_boundary(&iter, start,
 end))
+                       continue;
+
                 if (!sp) {
                         rcu_read_unlock();

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1629

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                         goto retry;
 
                 sp = NULL;
+               *split = true;
         }
 
         rcu_read_unlock();

This looks more reasonable for tdp_mmu_split_huge_pages_root();

Given that splitting only adds a new page to the paging structure (unlike page merging), I currently can't think of any current use cases that would be broken by the lack of TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock.

Quoted 12 lines

This is because:
1) if the split is triggered in a fault path, the hardware shouldn't have cached the old huge translation.
2) if the split is triggered in a zap or convert path,
- there shouldn't be concurrent faults on the range due to the protection of mmu_invalidate_range*.
- for concurrent splits on the same range, though the other vCPUs may temporally see stale huge TLB entries after they believe they have performed a split, they will be kicked off to flush the cache soon after tdp_mmu_split_huge_pages_root() returns in the first vCPU/host thread. This should be acceptable since I don't see any special guest needs that rely on pure splits.

Perhaps we should just go straight to the point:

Quoted 8 lines
Quoted 8 lines

What does "hugepage split" do, and what's the consequence of not flushing TLB.

Per make_small_spte(), the new child PTEs will carry all bits of hugepage PTE except they clear the 'hugepage bit (obviously)', and set the 'X' bit for NX hugepage thing.

That means if we leave the stale hugepage TLB, the CPU is still able to find the correct PFN and AFAICT there shouldn't be any other problem here.

The comments in tdp_mmu_split_huge_page() echo this.

    /*
     * Replace the huge spte with a pointer to the populated lower level
     * page table. Since we are making this change without a TLB flush vCPUs
     * will see a mix of the split mappings and the original huge mapping,
     * depending on what's currently in their TLB. This is fine from a
     * correctness standpoint since the translation will be the same either
     * way.
     */

Quoted 4 lines
Quoted 3 lines

For any fault
due to the stale hugepage TLB missing the 'X' permission, AFAICT KVM will just treat this as a spurious fault, which isn't nice but should have no harm.

Right, that isn't nice, though no harm.

According to SDM "Operations that Invalidate Cached Mappings":
The following operations invalidate cached mappings as indicated:
  - ...
  - An EPT violation invalidates any guest-physical mappings (associated with the current EPTRTA) that would be used to translate the guest-physical address that caused the EPT violation. If that guest-physical address was the translation of a linear address, the EPT violation also invalidates any combined mappings for that linear address associated with the current PCID, the current VPID and the current EPTRTA.
  - ...

If other CPUs have the stale hugepage TLB entry, there may be one spurious fault each.
Agree that it's not nice, but no harm.

Quoted 13 lines

Besides, I'm thinking of a scenario which is not currently existing though.

CPU 0 CPU 1
a1. split pages
a2. write protect pages
b1. split pages b2. write protect pages b3. start dirty page tracking
a3. flush TLB
a4. start dirty page tracking

Quoted 5 lines

If CPU 1 does not flush TLB after b2 (e.g., due to it finds the pages have been split and write protected by a1&a2), it will miss some dirty pages.

Currently CPU 1 always flush TLB before b3 unconditionally, so there's no problem.

Yes, for this write protection case, the TLB should be flushed. And currently, all (indirect) callers of tdp_mmu_split_huge_pages_root() do the TLB flush unconditionally.

For the TDX case, the TLB flush is done via the hook of secure EPT hugepage split.

It seems that the callers can decide whether the TLB flush is needed or not based on the following actions after hugepage split, may be with the info of whether split actually occurs or not.

Quoted 19 lines
Quoted 9 lines
Quoted 3 lines

So I tend to agree with your suggestion though the implementation in this patch is safer.

I am perhaps still missing something, as I am still trying to precisely understand in what cases you want to flush TLB when splitting hugepage.

I kinda tend to think you eventually want to flush TLB because eventually you want to _ZAP_. But needing to flush due to zap and needing to flush due to split is kinda different I think.

Though I currently couldn't find any use cases that depend on split alone, e.g. if there's any feature requiring the pages must be 4KB without any additional permission changes, I just wanted to make the code safer in case I missed any edge cases.

We surely don't want the window for CPUs to see huge pages and small pages lasts long.

Flushing TLB before releasing the mmu_lock allows other threads operating on the same range to see updated translations timely.

#71 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

In reply to Yan Zhao
Quoted 10 lines
Quoted 10 lines

- For non-TDX case ('only_cross_boundary == false'):

AFAICT the only user of tdp_mmu_split_huge_pages_root() is "eager hugepage splitting" during log-dirty. And per per the current implementation there are two callers of tdp_mmu_split_huge_pages_root():

kvm_mmu_try_split_huge_pages()
kvm_mmu_slot_try_split_huge_pages()

Quoted 7 lines
Quoted 2 lines

But they are both void functions which neither return whether flush TLB is needed, nor do TLB flush internally.

Actually callers of the two void functions do the TLB flush unconditionally in the end, i.e, in
kvm_mmu_slot_apply_flags(),
kvm_clear_dirty_log_protect(), and kvm_get_dirty_log_protect()).

Yeah, I didn't call this out.

Quoted 5 lines
Quoted 5 lines

So I am kinda confused.

Perhaps you mean for "shared memory of TDX guest", the caller will also pass 'only_cross_boundary == true' and the caller needs to perform TLB flush?

Quoted 8 lines

Sorry for the confusion.

Currently 'only_cross_boundary == true' is only for TDX private memory.

Returning flush is because kvm_split_cross_boundary_leafs() is potentially possible to be invoked for non-TDX cases as well in future (though currently it's only invoked for TDX alone). When that occurs, it's better to return flush to avoid the caller having to do flush unconditionally.

Exactly what "future" cases are you referring to?

Why do we need to consider it *NOW*?

Quoted 5 lines

Another reason is to keep consistency with tdp_mmu_zap_leafs(), which returns flush without differentiate whether the zap is for a mirror root not not. So, though kvm_mmu_remote_flush() on mirror root is not necessary, it's intentionally left for future optimization.

You mean non-blocking DEMOTE won't need to flush TLB internally when splitting but the caller needs to do the flush?

Anyway, all of above are not mentioned in the changelog. I think we need a clear explanation in the changelog to justify the change.

Quoted 84 lines
Quoted 62 lines

[...]

Quoted 42 lines
Quoted 28 lines

Something like below:

(quoted diff) +3 -1
@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct
 tdp_iter *iter,
 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                         struct kvm_mmu_page *root,
                                         gfn_t start, gfn_t end,
-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
 {
        struct kvm_mmu_page *sp = NULL;
        struct tdp_iter iter;
(quoted diff) +3 -0
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                if (!is_shadow_present_pte(iter.old_spte) ||
 !is_large_pte(iter.old_spte))
                        continue;
 
+               if (only_cross_boundary && !iter_cross_boundary(&iter, start,
 end))
+                       continue;
+
                if (!sp) {
                        rcu_read_unlock();

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1629

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                        goto retry;
 
                sp = NULL;
+               *split = true;
        }
 
        rcu_read_unlock();

This looks more reasonable for tdp_mmu_split_huge_pages_root();

Given that splitting only adds a new page to the paging structure (unlike page merging), I currently can't think of any current use cases that would be broken by the lack of TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock.

Quoted 12 lines

This is because:
1) if the split is triggered in a fault path, the hardware shouldn't have cached the old huge translation.
2) if the split is triggered in a zap or convert path,
- there shouldn't be concurrent faults on the range due to the protection of mmu_invalidate_range*.
- for concurrent splits on the same range, though the other vCPUs may temporally see stale huge TLB entries after they believe they have performed a split, they will be kicked off to flush the cache soon after tdp_mmu_split_huge_pages_root() returns in the first vCPU/host thread. This should be acceptable since I don't see any special guest needs that rely on pure splits.

Perhaps we should just go straight to the point:

What does "hugepage split" do, and what's the consequence of not flushing TLB.

Quoted 8 lines

Per make_small_spte(), the new child PTEs will carry all bits of hugepage PTE except they clear the 'hugepage bit (obviously)', and set the 'X' bit for NX hugepage thing.

That means if we leave the stale hugepage TLB, the CPU is still able to find the correct PFN and AFAICT there shouldn't be any other problem here. For any fault due to the stale hugepage TLB missing the 'X' permission, AFAICT KVM will just treat this as a spurious fault, which isn't nice but should have no harm.

Right, that isn't nice, though no harm.

Besides, I'm thinking of a scenario which is not currently existing though.

CPU 0 CPU 1
a1. split pages
a2. write protect pages
b1. split pages b2. write protect pages b3. start dirty page tracking
a3. flush TLB
a4. start dirty page tracking

Quoted 2 lines

If CPU 1 does not flush TLB after b2 (e.g., due to it finds the pages have been split and write protected by a1&a2), it will miss some dirty pages.

Do you have any actual concrete plan to foresee this is likely to happen in the future? E.g., why CPU1 wants to skip TLB flush after b2 due to a1&a2 etc?

To be honest I don't think we should discuss those hypothetical problems.

Quoted 19 lines

Currently CPU 1 always flush TLB before b3 unconditionally, so there's no problem.

Quoted 9 lines
Quoted 2 lines

So I tend to agree with your suggestion though the implementation in this patch is safer.

I am perhaps still missing something, as I am still trying to precisely understand in what cases you want to flush TLB when splitting hugepage.

I kinda tend to think you eventually want to flush TLB because eventually you want to _ZAP_. But needing to flush due to zap and needing to flush due to split is kinda different I think.

Though I currently couldn't find any use cases that depend on split alone, e.g. if there's any feature requiring the pages must be 4KB without any additional permission changes, I just wanted to make the code safer in case I missed any edge cases.

Quoted 5 lines

We surely don't want the window for CPUs to see huge pages and small pages lasts long.

Flushing TLB before releasing the mmu_lock allows other threads operating on the same range to see updated translations timely.

In the upstream code most callers of tdp_mmu_iter_cond_resched() call it w/o flushing TLB when yield happens, so the "window of stale TLB" already exists -- it's just not stale hugepage TLBs, but other stale TLBs.

But I agree it's not good to have stale TLBs, and looking at recover_huge_pages_range(), it also does TLB flush when yielding if there's already hugepage merge happened.

So if you want to make tdp_mmu_split_huge_pages_root() handle TLB flush, perhaps we can make it like recover_huge_pages_range(). But AFAICT we also want to make tdp_mmu_split_huge_pages_root() return whether flush is needed, but not actually perform TLB flush for non-yielding case, because otherwise we need to revisit the log-dirty code to avoid duplicated TLB flush.

And then the 'only_cross_boundary' can be added to it.

Btw, a second thought on the 'only_cross_boundary':

My first glance of 'only_cross_boundary' was it's a little bit odd, because you actually only need to split the hugepage where 'start' and 'end' is in middle of a hugepage.

So alternatively, instead of yet adding another 'only_cross_boundary' to tdp_mmu_split_huge_pages_root(), I think we can also make the caller check the range and only call tdp_mmu_split_huge_pages_root() when the range crosses the hugepage boundary?

E.g., for a range [1G, 2G), it's doesn't cross any 2M boundary, thus the caller can skip calling tdp_mmu_split_huge_pages_root(). If the range is [1G + 1M, 2G), then the caller can know only the first [1G, 1G + 2M) needs splitting. This also saves unnecessary iter walk for the rest range [1G + 2M, 2G).

I think if we only consider 2M hugepage but not 1G page, then it should not be that complicated to check the range and only call tdp_mmu_split_huge_pages_root() for the range that is truly needs splitting?

author
#72 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Hi Kai and all,

Let me summarize my points clearly in advance:
(I guess I failed to do it explicitly in my previous mails [1][2]).

- I agree with Kai's suggestion to return a "bool *split" to callers of kvm_split_cross_boundary_leafs(). The callers can choose to do TLB flush or not, since we don't want them to do TLB flush unconditionally. (see the "Note" below).

- I think it's OK to skip TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock in tdp_mmu_split_huge_pages_root(), as there's no known use case impacted up to now, according to the analysis in [1].

- Invoke kvm_flush_remote_tlbs() for tdp_mmu_split_huge_pages_root() in this series is for
a) code completeness.
kvm_split_cross_boundary_leafs() does not force that the root must be a mirror root.

TDX alone doesn't require invoking kvm_flush_remote_tlbs() as it's done implicitly in tdx_sept_split_private_spt(). TDX share memory also does not invoke kvm_split_cross_boundary_leafs().

b) code consistency.
kvm_unmap_gfn_range() also returns flush for callers to invoke kvm_flush_remote_tlbs(), even when the range is of KVM_FILTER_PRIVATE alone.

I'll update the patch with proper comments to explain the above points if you are agreed.

Thanks
Yan

Note:
Currently there are 3 callers of kvm_split_cross_boundary_leafs():
1) tdx_check_accept_level(), which actually has no need to invoke kvm_flush_remote_tlbs() since it splits mirror root only.

2) kvm_arch_pre_set_memory_attributes(), which can combine the flush together with the TLB flush due to kvm_unmap_gfn_range().

3) kvm_gmem_split_private(), which is invoked by gmem punch_hole and gmem conversion from private to shared. The caller can choose to do TLB flush separately or together with kvm_gmem_zap() later.

[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]

Quoted 1 line · Kai
Quoted 1 line
Quoted 1 line

Will reply the rest of your mail seperately later.

author
#73 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

Quoted 9 lines · Yan Zhao

Hi Kai and all,

Let me summarize my points clearly in advance:
(I guess I failed to do it explicitly in my previous mails [1][2]).

- I agree with Kai's suggestion to return a "bool *split" to callers of kvm_split_cross_boundary_leafs(). The callers can choose to do TLB flush or not, since we don't want them to do TLB flush unconditionally. (see the "Note" below).

Hi Kai,

Thanks for your review and bringing up the TLB flush issue!

After further thought, I finally chose not to return the split status in kvm_split_cross_boundary_leafs(), because the split status is not accurate given that we don't flush TLB before releasing mmu_lock in tdp_mmu_split_huge_pages_root(). i.e., when the function returns split as false, splits could still have occurred during the temporary release of mmu_lock.

So, I implemented the API like this:
(1) Do not return split status in kvm_split_cross_boundary_leafs().
(2) Let the caller decide whether and how to flush TLB according to the use cases. e.g.,
- if it's for dirty tracking (e.g., splits before turning on PML), unconditionally flush TLB.
- if it's in the fault path, e.g., tdx_check_accept_level(). No TLB flush is required (current TDX's tdx_track() also ensures no need for a separate flush).
- if it's for gmem punch hole or page conversions, the callers can delay the TLB flush for splits and combine it with the flush for zaps.

I've posted this implementation in v3
https://lore.kernel.org/all/[email protected].

Please let me know if it doesn't look good.

Thanks
Yan

Quoted 44 lines

- I think it's OK to skip TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock in tdp_mmu_split_huge_pages_root(), as there's no known use case impacted up to now, according to the analysis in [1].

- Invoke kvm_flush_remote_tlbs() for tdp_mmu_split_huge_pages_root() in this series is for
a) code completeness.
kvm_split_cross_boundary_leafs() does not force that the root must be a mirror root.

TDX alone doesn't require invoking kvm_flush_remote_tlbs() as it's done implicitly in tdx_sept_split_private_spt(). TDX share memory also does not invoke kvm_split_cross_boundary_leafs().

b) code consistency.
kvm_unmap_gfn_range() also returns flush for callers to invoke kvm_flush_remote_tlbs(), even when the range is of KVM_FILTER_PRIVATE alone.

I'll update the patch with proper comments to explain the above points if you are agreed.

Thanks
Yan

Note:
Currently there are 3 callers of kvm_split_cross_boundary_leafs():
1) tdx_check_accept_level(), which actually has no need to invoke kvm_flush_remote_tlbs() since it splits mirror root only.

2) kvm_arch_pre_set_memory_attributes(), which can combine the flush together with the TLB flush due to kvm_unmap_gfn_range().

3) kvm_gmem_split_private(), which is invoked by gmem punch_hole and gmem conversion from private to shared. The caller can choose to do TLB flush separately or together with kvm_gmem_zap() later.

[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]

Quoted 1 line · Kai
Quoted 1 line
Quoted 1 line

Will reply the rest of your mail seperately later.

author
#74 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

In reply to Huang, Kai
Quoted 8 lines · Kai
Quoted 8 lines
Quoted 4 lines

So I am kinda confused.

Perhaps you mean for "shared memory of TDX guest", the caller will also pass 'only_cross_boundary == true' and the caller needs to perform TLB flush?

Sorry for the confusion.

Currently 'only_cross_boundary == true' is only for TDX private memory.

Quoted 8 lines
Quoted 4 lines

Returning flush is because kvm_split_cross_boundary_leafs() is potentially possible to be invoked for non-TDX cases as well in future (though currently it's only invoked for TDX alone). When that occurs, it's better to return flush to avoid the caller having to do flush unconditionally.

Exactly what "future" cases are you referring to?

Why do we need to consider it *NOW*?

The API kvm_split_cross_boundary_leafs() does not force that the root must be a mirror root. So, it's better to return "split" status and let the caller decide whether to invoke kvm_flush_remote_tlbs(). This is for code completeness and consistency as explained in [1].

[1] https://lore.kernel.org/all/aR08f%[email protected]/

Quoted 8 lines
Quoted 5 lines

Another reason is to keep consistency with tdp_mmu_zap_leafs(), which returns flush without differentiate whether the zap is for a mirror root not not. So, though kvm_mmu_remote_flush() on mirror root is not necessary, it's intentionally left for future optimization.

You mean non-blocking DEMOTE won't need to flush TLB internally when splitting but the caller needs to do the flush?

I mean as an API implemented in KVM core, it's better to have kvm_split_cross_boundary_leafs() to make any assumption on whether TDX internally have performed the TLB flush or whether non-blocking DEMOTE needs flush.

We can return "split" and let callers decide flush or not.

The optimization to avoid invoking kvm_flush_remote_tlbs() for zaps/splits on a mirror root alone can be implemented in the future when necessary.

Quoted 2 lines

Anyway, all of above are not mentioned in the changelog. I think we need a clear explanation in the changelog to justify the change.

Will do.

Quoted 83 lines
Quoted 83 lines
Quoted 61 lines

[...]

Quoted 42 lines
Quoted 28 lines

Something like below:

(quoted diff) +3 -1
@@ -1558,7 +1558,9 @@ static int tdp_mmu_split_huge_page(struct kvm *kvm, struct
 tdp_iter *iter,
 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                         struct kvm_mmu_page *root,
                                         gfn_t start, gfn_t end,
-                                        int target_level, bool shared)
+                                        int target_level, bool shared,
+                                        bool only_cross_boundary,
+                                        bool *split)
 {
        struct kvm_mmu_page *sp = NULL;
        struct tdp_iter iter;
(quoted diff) +3 -0
@@ -1584,6 +1586,9 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                if (!is_shadow_present_pte(iter.old_spte) ||
 !is_large_pte(iter.old_spte))
                        continue;
 
+               if (only_cross_boundary && !iter_cross_boundary(&iter, start,
 end))
+                       continue;
+
                if (!sp) {
                        rcu_read_unlock();

arch/x86/kvm/mmu/tdp_mmu.c static int tdp_mmu_split_huge_pages_root(struct kvm *kvm, · line 1629

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
@@ -1618,6 +1623,7 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                        goto retry;
 
                sp = NULL;
+               *split = true;
        }
 
        rcu_read_unlock();

This looks more reasonable for tdp_mmu_split_huge_pages_root();

Given that splitting only adds a new page to the paging structure (unlike page merging), I currently can't think of any current use cases that would be broken by the lack of TLB flush before tdp_mmu_iter_cond_resched() releases the mmu_lock.

Quoted 12 lines

This is because:
1) if the split is triggered in a fault path, the hardware shouldn't have cached the old huge translation.
2) if the split is triggered in a zap or convert path,
- there shouldn't be concurrent faults on the range due to the protection of mmu_invalidate_range*.
- for concurrent splits on the same range, though the other vCPUs may temporally see stale huge TLB entries after they believe they have performed a split, they will be kicked off to flush the cache soon after tdp_mmu_split_huge_pages_root() returns in the first vCPU/host thread. This should be acceptable since I don't see any special guest needs that rely on pure splits.

Perhaps we should just go straight to the point:

What does "hugepage split" do, and what's the consequence of not flushing TLB.

Quoted 8 lines

Per make_small_spte(), the new child PTEs will carry all bits of hugepage PTE except they clear the 'hugepage bit (obviously)', and set the 'X' bit for NX hugepage thing.

That means if we leave the stale hugepage TLB, the CPU is still able to find the correct PFN and AFAICT there shouldn't be any other problem here. For any fault due to the stale hugepage TLB missing the 'X' permission, AFAICT KVM will just treat this as a spurious fault, which isn't nice but should have no harm.

Right, that isn't nice, though no harm.

Besides, I'm thinking of a scenario which is not currently existing though.

CPU 0 CPU 1
a1. split pages
a2. write protect pages
b1. split pages b2. write protect pages b3. start dirty page tracking
a3. flush TLB
a4. start dirty page tracking

Quoted 7 lines
Quoted 2 lines

If CPU 1 does not flush TLB after b2 (e.g., due to it finds the pages have been split and write protected by a1&a2), it will miss some dirty pages.

Do you have any actual concrete plan to foresee this is likely to happen in the future? E.g., why CPU1 wants to skip TLB flush after b2 due to a1&a2 etc?

To be honest I don't think we should discuss those hypothetical problems.

Sorry about the confusion.

I just wanted to express why I thought it's safer to do flush before releasing mmu_lock. However, as I mentioned in the previous replies, I can't find any current use cases impacted by skipping this flush. So I think it's ok not to flush before releasing mmu_lock.

Will update the patch comment to explain why the skipping of flush is ok. (I think current upstream tdp_mmu_split_huge_pages_root() lacks a comment of why it's safe not to do the flush before releasing mmu_lock).

Quoted 42 lines
Quoted 18 lines

Currently CPU 1 always flush TLB before b3 unconditionally, so there's no problem.

Quoted 9 lines
Quoted 2 lines

So I tend to agree with your suggestion though the implementation in this patch is safer.

I am perhaps still missing something, as I am still trying to precisely understand in what cases you want to flush TLB when splitting hugepage.

I kinda tend to think you eventually want to flush TLB because eventually you want to _ZAP_. But needing to flush due to zap and needing to flush due to split is kinda different I think.

Though I currently couldn't find any use cases that depend on split alone, e.g. if there's any feature requiring the pages must be 4KB without any additional permission changes, I just wanted to make the code safer in case I missed any edge cases.

Quoted 5 lines

We surely don't want the window for CPUs to see huge pages and small pages lasts long.

Flushing TLB before releasing the mmu_lock allows other threads operating on the same range to see updated translations timely.

In the upstream code most callers of tdp_mmu_iter_cond_resched() call it w/o flushing TLB when yield happens, so the "window of stale TLB" already exists -- it's just not stale hugepage TLBs, but other stale TLBs.

But I agree it's not good to have stale TLBs, and looking at recover_huge_pages_range(), it also does TLB flush when yielding if there's already hugepage merge happened.

So if you want to make tdp_mmu_split_huge_pages_root() handle TLB flush, perhaps we can make it like recover_huge_pages_range(). But AFAICT we also want to make tdp_mmu_split_huge_pages_root() return whether flush is needed, but not actually perform TLB flush for non-yielding case, because otherwise we need to revisit the log-dirty code to avoid duplicated TLB flush.

And then the 'only_cross_boundary' can be added to it.

Btw, a second thought on the 'only_cross_boundary':

Quoted 8 lines

My first glance of 'only_cross_boundary' was it's a little bit odd, because you actually only need to split the hugepage where 'start' and 'end' is in middle of a hugepage.

So alternatively, instead of yet adding another 'only_cross_boundary' to tdp_mmu_split_huge_pages_root(), I think we can also make the caller check the range and only call tdp_mmu_split_huge_pages_root() when the range crosses the hugepage boundary?

I don't think it's good.

- The code to check if a range crosses hugepage boundary is cumbersome and level dependent. As you point out below, [1G, 1G + 2M) does not need splitting if there's only 2M mappings, but it needs splitting when there're 1G mappings.

For an API implemented in KVM core, it's better not to assume there're no 1G mappings. Not to mention TDX itself will support 1G in future.

- When a range is determined as "truly needs splitting", e.g. [1G-8K, 1G+8K), tdp_mmu_split_huge_pages_root() still needs to return 'split' status since splitting may not occur due to no present mappings or no 2M mappings.

Quoted 8 lines

E.g., for a range [1G, 2G), it's doesn't cross any 2M boundary, thus the caller can skip calling tdp_mmu_split_huge_pages_root(). If the range is [1G + 1M, 2G), then the caller can know only the first [1G, 1G + 2M) needs splitting. This also saves unnecessary iter walk for the rest range [1G + 2M, 2G).

I think if we only consider 2M hugepage but not 1G page, then it should not be that complicated to check the range and only call tdp_mmu_split_huge_pages_root() for the range that is truly needs splitting?

author
#75 lore raw

Re: [RFC PATCH v2 12/23] KVM: x86/mmu: Introduce kvm_split_cross_boundary_leafs()

In reply to Yan Zhao
Quoted 40 lines · Yan Zhao
arch/x86/kvm/mmu/mmu.c +26 -0
index 9182192daa3a..13910ae05f76 100644
@@ -1647,6 +1647,33 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm,
 				 start, end - 1, can_yield, true, flush);
 }
 
+/*
+ * Split large leafs crossing the boundary of the specified range
+ *
+ * Return value:
+ * 0 : success, no flush is required;
+ * 1 : success, flush is required;
+ * <0: failure.
+ */
+int kvm_split_cross_boundary_leafs(struct kvm *kvm, struct kvm_gfn_range *range,
+				   bool shared)
+{
+	bool ret = 0;
+
+	lockdep_assert_once(kvm->mmu_invalidate_in_progress ||
+			    lockdep_is_held(&kvm->slots_lock) ||
+			    srcu_read_lock_held(&kvm->srcu));
+
+	if (!range->may_block)
+		return -EOPNOTSUPP;
+
+	if (tdp_mmu_enabled)
+		ret = kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(kvm, range, shared);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(kvm_split_cross_boundary_leafs);
 
include/linux/kvm_host.h +0 -0
index fb79d2b7decd..6137b76341e1 100644

include/linux/kvm_host.h struct kvm_gfn_range { · line 277

Quoted 5 lines
include/linux/kvm_host.h +2 -0
@@ -273,6 +273,8 @@ struct kvm_gfn_range {
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
+int kvm_split_cross_boundary_leafs(struct kvm *kvm, struct kvm_gfn_range *range,
+				   bool shared);

Note to myself: Provide a default implementation for non-x86 platforms.

author
#76 lore raw

RFC PATCH v2 13/23 KVM: x86: Introduce hugepage_set_guest_inhibit()

In reply to Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

To facilitate passing the guest's accept level information to the KVM MMU core and to prevent the repeated mapping of a GFN at different levels due to different accept levels specified by different vCPUs, introduce the interface hugepage_set_guest_inhibit(). This interface specifies across vCPUs that mapping at a certain level is inhibited from the guest.

The KVM_LPAGE_GUEST_INHIBIT_FLAG bit is currently modified in one direction (set), so no clear interface is provided.

---
RFC v2:
- new in RFC v2

---
 arch/x86/kvm/mmu.h     |  3 +++
 arch/x86/kvm/mmu/mmu.c | 21 ++++++++++++++++++---
 2 files changed, 21 insertions(+), 3 deletions(-)
arch/x86/kvm/mmu.h +3 -0
index b122255c7d4e..c2d8819f3438 100644
@@ -326,4 +326,7 @@ static inline bool kvm_is_gfn_alias(struct kvm *kvm, gfn_t gfn)
 {
 	return gfn & kvm_gfn_direct_bits(kvm);
 }
+
+void hugepage_set_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level);
+bool hugepage_test_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level);
 #endif
arch/x86/kvm/mmu/mmu.c +18 -3
index 13910ae05f76..1c639286aac2 100644
@@ -721,12 +721,14 @@ static struct kvm_lpage_info *lpage_info_slot(gfn_t gfn,
 }
 
 /*
- * The most significant bit in disallow_lpage tracks whether or not memory
- * attributes are mixed, i.e. not identical for all gfns at the current level.
+ * The most 2 significant bits in disallow_lpage tracks whether or not memory
+ * attributes are mixed, i.e. not identical for all gfns at the current level,
+ * or whether or not guest inhibits the current level of hugepage at the gfn.
  * The lower order bits are used to refcount other cases where a hugepage is
  * disallowed, e.g. if KVM has shadow a page table at the gfn.
  */
 #define KVM_LPAGE_MIXED_FLAG	BIT(31)
+#define KVM_LPAGE_GUEST_INHIBIT_FLAG   BIT(30)
 
 static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
 					    gfn_t gfn, int count)
@@ -739,7 +741,8 @@ static void update_gfn_disallow_lpage_count(const struct kvm_memory_slot *slot,
 
 		old = linfo->disallow_lpage;
 		linfo->disallow_lpage += count;
-		WARN_ON_ONCE((old ^ linfo->disallow_lpage) & KVM_LPAGE_MIXED_FLAG);
+		WARN_ON_ONCE((old ^ linfo->disallow_lpage) &
+			     (KVM_LPAGE_MIXED_FLAG | KVM_LPAGE_GUEST_INHIBIT_FLAG));
 	}
 }
 
@@ -1647,6 +1650,18 @@ static bool __kvm_rmap_zap_gfn_range(struct kvm *kvm,
 				 start, end - 1, can_yield, true, flush);
 }
 
+bool hugepage_test_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level)
+{
+	return lpage_info_slot(gfn, slot, level)->disallow_lpage & KVM_LPAGE_GUEST_INHIBIT_FLAG;
+}
+EXPORT_SYMBOL_GPL(hugepage_test_guest_inhibit);
+
+void hugepage_set_guest_inhibit(struct kvm_memory_slot *slot, gfn_t gfn, int level)
+{
+	lpage_info_slot(gfn, slot, level)->disallow_lpage |= KVM_LPAGE_GUEST_INHIBIT_FLAG;
+}
+EXPORT_SYMBOL_GPL(hugepage_set_guest_inhibit);
+
 /*
  * Split large leafs crossing the boundary of the specified range
  *
Signature · 1 line
2.43.2
author
#77 lore raw

RFC PATCH v2 14/23 KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

In reply to Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

The TDX module thus enables the EPT violation VMExit to carry the guest's accept level when the VMExit is caused by the guest's accept operation.

Therefore, in TDX's EPT violation handler
(1) Set the guest inhibit bit in the lpage info to prevent KVM MMU core from mapping at a higher a level than the guest's accept level.

(2) Split any existing huge mapping at the fault GFN to avoid unsupported splitting under the shared mmu_lock by TDX.

Use write mmu_lock to pretect (1) and (2) for now. If future KVM TDX can perform the actual splitting under shared mmu_lock with enhanced TDX modules, (1) is possible to be called under shared mmu_lock, and (2) would become unnecessary.

As an optimization, this patch calls hugepage_test_guest_inhibit() without holding the mmu_lock to reduce the frequency of acquiring the write mmu_lock. The write mmu_lock is thus only acquired if the guest inhibit bit is not already set. This is safe because the guest inhibit bit is set in a one-way manner while the splitting under the write mmu_lock is performed before setting the guest inhibit bit.

---
RFC v2
- Change tdx_get_accept_level() to tdx_check_accept_level().
- Invoke kvm_split_cross_boundary_leafs() and hugepage_set_guest_inhibit() to change KVM mapping level in a global way according to guest accept level. (Rick, Sean).

RFC v1:
- Introduce tdx_get_accept_level() to get guest accept level.
- Use tdx->violation_request_level and tdx->violation_gfn* to pass guest accept level to tdx_gmem_private_max_mapping_level() to detemine KVM mapping level.

---
 arch/x86/kvm/vmx/tdx.c      | 50 +++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx_arch.h |  3 +++
 2 files changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +50 -0
index 035d81275be4..71115058e5e6 100644
@@ -2019,6 +2019,53 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
 	return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
 }
 
+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(vcpu->kvm, gfn);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct kvm *kvm = vcpu->kvm;
+	u64 eeq_type, eeq_info;
+	int level = -1;
+
+	if (!slot)
+		return 0;
+
+	eeq_type = tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK;
+	if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_ACCEPT)
+		return 0;
+
+	eeq_info = (tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+		   TDX_EXT_EXIT_QUAL_INFO_SHIFT;
+
+	level = (eeq_info & GENMASK(2, 0)) + 1;
+
+	if (level == PG_LEVEL_4K || level == PG_LEVEL_2M) {
+		if (!hugepage_test_guest_inhibit(slot, gfn, level + 1)) {
+			gfn_t base_gfn = gfn_round_for_level(gfn, level);
+			struct kvm_gfn_range gfn_range = {
+				.start = base_gfn,
+				.end = base_gfn + KVM_PAGES_PER_HPAGE(level),
+				.slot = slot,
+				.may_block = true,
+				.attr_filter = KVM_FILTER_PRIVATE,
+			};
+
+			scoped_guard(write_lock, &kvm->mmu_lock) {
+				int ret;
+
+				ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+				if (ret)
+					return ret;
+
+				hugepage_set_guest_inhibit(slot, gfn, level + 1);
+				if (level == PG_LEVEL_4K)
+					hugepage_set_guest_inhibit(slot, gfn, level + 2);
+			}
+		}
+	}
+	return 0;
+}
+
 static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
 {
 	unsigned long exit_qual;
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
 		 */
 		exit_qual = EPT_VIOLATION_ACC_WRITE;
 
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+
 		/* Only private GPA triggers zero-step mitigation */
 		local_retry = true;
 	} else {
arch/x86/kvm/vmx/tdx_arch.h +3 -0
index a30e880849e3..af006a73ee05 100644
@@ -82,7 +82,10 @@ struct tdx_cpuid_value {
 #define TDX_TD_ATTR_PERFMON		BIT_ULL(63)
 
 #define TDX_EXT_EXIT_QUAL_TYPE_MASK	GENMASK(3, 0)
+#define TDX_EXT_EXIT_QUAL_TYPE_ACCEPT  1
 #define TDX_EXT_EXIT_QUAL_TYPE_PENDING_EPT_VIOLATION  6
+#define TDX_EXT_EXIT_QUAL_INFO_MASK	GENMASK(63, 32)
+#define TDX_EXT_EXIT_QUAL_INFO_SHIFT	32
 /*
  * TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B.
  */
Signature · 1 line
2.43.2
#78 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 15 lines · Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

The TDX module thus enables the EPT violation VMExit to carry the guest's accept level when the VMExit is caused by the guest's accept operation.

Therefore, in TDX's EPT violation handler
(1) Set the guest inhibit bit in the lpage info to prevent KVM MMU core from mapping at a higher a level than the guest's accept level.

Quoted 7 lines

(2) Split any existing huge mapping at the fault GFN to avoid unsupported splitting under the shared mmu_lock by TDX.

Use write mmu_lock to pretect (1) and (2) for now. If future KVM TDX can perform the actual splitting under shared mmu_lock with enhanced TDX modules, (1) is possible to be called under shared mmu_lock, and (2) would become unnecessary.

The description for (1) and (2) reversed?

Quoted 64 lines

As an optimization, this patch calls hugepage_test_guest_inhibit() without holding the mmu_lock to reduce the frequency of acquiring the write mmu_lock. The write mmu_lock is thus only acquired if the guest inhibit bit is not already set. This is safe because the guest inhibit bit is set in a one-way manner while the splitting under the write mmu_lock is performed before setting the guest inhibit bit.

---
RFC v2
- Change tdx_get_accept_level() to tdx_check_accept_level().
- Invoke kvm_split_cross_boundary_leafs() and hugepage_set_guest_inhibit() to change KVM mapping level in a global way according to guest accept level. (Rick, Sean).

RFC v1:
- Introduce tdx_get_accept_level() to get guest accept level.
- Use tdx->violation_request_level and tdx->violation_gfn* to pass guest accept level to tdx_gmem_private_max_mapping_level() to detemine KVM mapping level.

---
  arch/x86/kvm/vmx/tdx.c      | 50 +++++++++++++++++++++++++++++++++++++
  arch/x86/kvm/vmx/tdx_arch.h |  3 +++
  2 files changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +27 -0
index 035d81275be4..71115058e5e6 100644
@@ -2019,6 +2019,53 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
  	return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
  }
 
+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(vcpu->kvm, gfn);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct kvm *kvm = vcpu->kvm;
+	u64 eeq_type, eeq_info;
+	int level = -1;
+
+	if (!slot)
+		return 0;
+
+	eeq_type = tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK;
+	if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_ACCEPT)
+		return 0;
+
+	eeq_info = (tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+		   TDX_EXT_EXIT_QUAL_INFO_SHIFT;
+
+	level = (eeq_info & GENMASK(2, 0)) + 1;
+
+	if (level == PG_LEVEL_4K || level == PG_LEVEL_2M) {
+		if (!hugepage_test_guest_inhibit(slot, gfn, level + 1)) {
+			gfn_t base_gfn = gfn_round_for_level(gfn, level);
+			struct kvm_gfn_range gfn_range = {
+				.start = base_gfn,
+				.end = base_gfn + KVM_PAGES_PER_HPAGE(level),
+				.slot = slot,

arch/x86/kvm/vmx/tdx.c static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp · line 2058

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+				.may_block = true,
+				.attr_filter = KVM_FILTER_PRIVATE,
+			};
+
+			scoped_guard(write_lock, &kvm->mmu_lock) {
+				int ret;
+
+				ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+				if (ret)
+					return ret;

kvm_split_cross_boundary_leafs() calls kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(), which could return flush as 1 if any of the huge page crossing boundary is split, return directly when ret is non-zero seems not right. Also, the TLB flush should also be taken care because in kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(), TLB flush is only done for negative return value.

Quoted 38 lines
arch/x86/kvm/vmx/tdx.c +13 -0
+
+				hugepage_set_guest_inhibit(slot, gfn, level + 1);
+				if (level == PG_LEVEL_4K)
+					hugepage_set_guest_inhibit(slot, gfn, level + 2);
+			}
+		}
+	}
+	return 0;
+}
+
  static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  {
  	unsigned long exit_qual;
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;
 
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+
  		/* Only private GPA triggers zero-step mitigation */
  		local_retry = true;
  	} else {
arch/x86/kvm/vmx/tdx_arch.h +3 -0
index a30e880849e3..af006a73ee05 100644
@@ -82,7 +82,10 @@ struct tdx_cpuid_value {
  #define TDX_TD_ATTR_PERFMON		BIT_ULL(63)
 
  #define TDX_EXT_EXIT_QUAL_TYPE_MASK	GENMASK(3, 0)
+#define TDX_EXT_EXIT_QUAL_TYPE_ACCEPT  1
  #define TDX_EXT_EXIT_QUAL_TYPE_PENDING_EPT_VIOLATION  6
+#define TDX_EXT_EXIT_QUAL_INFO_MASK	GENMASK(63, 32)
+#define TDX_EXT_EXIT_QUAL_INFO_SHIFT	32
  /*
   * TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B.
   */
author
#79 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 21 lines · Binbin Wu
Quoted 18 lines · Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

The TDX module thus enables the EPT violation VMExit to carry the guest's accept level when the VMExit is caused by the guest's accept operation.

Therefore, in TDX's EPT violation handler
(1) Set the guest inhibit bit in the lpage info to prevent KVM MMU core from mapping at a higher a level than the guest's accept level.

(2) Split any existing huge mapping at the fault GFN to avoid unsupported splitting under the shared mmu_lock by TDX.

Quoted 6 lines
Quoted 4 lines

Use write mmu_lock to pretect (1) and (2) for now. If future KVM TDX can perform the actual splitting under shared mmu_lock with enhanced TDX modules, (1) is possible to be called under shared mmu_lock, and (2) would become unnecessary.

The description for (1) and (2) reversed?

No.
After supporting splitting under shared mmu_lock,
- setting guest inhibit bit can be performed under shared mmu_lock. (*)
- splitting existing huge mapping under write mmu_lock here would be unnecessary.

(*) is still required to convey the info of which max level the guest requires.
(as explained in "Open 1: How to pass guest's ACCEPT level info" in the cover letter).

Quoted 66 lines
Quoted 66 lines

As an optimization, this patch calls hugepage_test_guest_inhibit() without holding the mmu_lock to reduce the frequency of acquiring the write mmu_lock. The write mmu_lock is thus only acquired if the guest inhibit bit is not already set. This is safe because the guest inhibit bit is set in a one-way manner while the splitting under the write mmu_lock is performed before setting the guest inhibit bit.

---
RFC v2
- Change tdx_get_accept_level() to tdx_check_accept_level().
- Invoke kvm_split_cross_boundary_leafs() and hugepage_set_guest_inhibit() to change KVM mapping level in a global way according to guest accept level. (Rick, Sean).

RFC v1:
- Introduce tdx_get_accept_level() to get guest accept level.
- Use tdx->violation_request_level and tdx->violation_gfn* to pass guest accept level to tdx_gmem_private_max_mapping_level() to detemine KVM mapping level.

---
  arch/x86/kvm/vmx/tdx.c      | 50 +++++++++++++++++++++++++++++++++++++
  arch/x86/kvm/vmx/tdx_arch.h |  3 +++
  2 files changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +31 -0
index 035d81275be4..71115058e5e6 100644
@@ -2019,6 +2019,53 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
  	return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
  }
+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(vcpu->kvm, gfn);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct kvm *kvm = vcpu->kvm;
+	u64 eeq_type, eeq_info;
+	int level = -1;
+
+	if (!slot)
+		return 0;
+
+	eeq_type = tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK;
+	if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_ACCEPT)
+		return 0;
+
+	eeq_info = (tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+		   TDX_EXT_EXIT_QUAL_INFO_SHIFT;
+
+	level = (eeq_info & GENMASK(2, 0)) + 1;
+
+	if (level == PG_LEVEL_4K || level == PG_LEVEL_2M) {
+		if (!hugepage_test_guest_inhibit(slot, gfn, level + 1)) {
+			gfn_t base_gfn = gfn_round_for_level(gfn, level);
+			struct kvm_gfn_range gfn_range = {
+				.start = base_gfn,
+				.end = base_gfn + KVM_PAGES_PER_HPAGE(level),
+				.slot = slot,
+				.may_block = true,
+				.attr_filter = KVM_FILTER_PRIVATE,
+			};
+
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp · line 2058

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+			scoped_guard(write_lock, &kvm->mmu_lock) {
+				int ret;
+
+				ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+				if (ret)
+					return ret;

kvm_split_cross_boundary_leafs() calls kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(), which could return flush as 1 if any of the huge page crossing boundary is split, return directly when ret is non-zero seems not right. Also, the TLB flush should also be taken care because in kvm_tdp_mmu_gfn_range_split_cross_boundary_leafs(), TLB flush is only done for negative return value.

Oh, good catch!

I forgot about the 2 facts. Will fix them.

Quoted 37 lines
Quoted 36 lines
arch/x86/kvm/vmx/tdx.c +13 -0
+
+				hugepage_set_guest_inhibit(slot, gfn, level + 1);
+				if (level == PG_LEVEL_4K)
+					hugepage_set_guest_inhibit(slot, gfn, level + 2);
+			}
+		}
+	}
+	return 0;
+}
+
  static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  {
  	unsigned long exit_qual;
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+
  		/* Only private GPA triggers zero-step mitigation */
  		local_retry = true;
  	} else {
arch/x86/kvm/vmx/tdx_arch.h +3 -0
index a30e880849e3..af006a73ee05 100644
@@ -82,7 +82,10 @@ struct tdx_cpuid_value {
  #define TDX_TD_ATTR_PERFMON		BIT_ULL(63)
  #define TDX_EXT_EXIT_QUAL_TYPE_MASK	GENMASK(3, 0)
+#define TDX_EXT_EXIT_QUAL_TYPE_ACCEPT  1
  #define TDX_EXT_EXIT_QUAL_TYPE_PENDING_EPT_VIOLATION  6
+#define TDX_EXT_EXIT_QUAL_INFO_MASK	GENMASK(63, 32)
+#define TDX_EXT_EXIT_QUAL_INFO_SHIFT	32
  /*
   * TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B.
   */
#80 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

In reply to Yan Zhao

arch/x86/kvm/vmx/tdx.c static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) · line 2096

Quoted 7 lines · Yan Zhao
arch/x86/kvm/vmx/tdx.c +0 -0
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;

arch/x86/kvm/vmx/tdx.c +3 -0
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+

I don't think you should return RET_PF_RETRY here.

This is still at very early stage of EPT violation. The caller of tdx_handle_ept_violation() is expecting either 0, 1, or negative error code.

author
#81 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 11 lines · Kai

arch/x86/kvm/vmx/tdx.c static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) · line 2096

Quoted 7 lines · Yan Zhao
arch/x86/kvm/vmx/tdx.c +0 -0
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;

arch/x86/kvm/vmx/tdx.c +3 -0
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+

I don't think you should return RET_PF_RETRY here.

Quoted 2 lines

This is still at very early stage of EPT violation. The caller of tdx_handle_ept_violation() is expecting either 0, 1, or negative error code.

Hmm, strictly speaking, the caller of the EPT violation handler is expecting 0, >0, or negative error code.

vcpu_run

  |->r = vcpu_enter_guest(vcpu);
  |        |->r = kvm_x86_call(handle_exit)(vcpu, exit_fastpath);
  |        |  return r;
  |  if (r <= 0)
  |     break;

handle_ept_violation
|->return __vmx_handle_ept_violation(vcpu, gpa, exit_qualification);

tdx_handle_ept_violation
|->ret = __vmx_handle_ept_violation(vcpu, gpa, exit_qual); | return ret;

The current VMX/TDX's EPT violation handlers returns RET_PF_* to the caller since commit 7c5480386300 ("KVM: x86/mmu: Return RET_PF* instead of 1 in kvm_mmu_page_fault") for the sake of zero-step mitigation.

This is no problem, because

enum {
RET_PF_CONTINUE = 0,
RET_PF_RETRY,
RET_PF_EMULATE,
RET_PF_WRITE_PROTECTED,
RET_PF_INVALID,
RET_PF_FIXED,
RET_PF_SPURIOUS,
};

/*
* Define RET_PF_CONTINUE as 0 to allow for
* - efficient machine code when checking for CONTINUE, e.g.
* "TEST %rax, %rax, JNZ", as all "stop!" values are non-zero,
* - kvm_mmu_do_page_fault() to return other RET_PF_* as a positive value. */
static_assert(RET_PF_CONTINUE == 0);

#82 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 47 lines · Yan Zhao
Quoted 11 lines · Kai

arch/x86/kvm/vmx/tdx.c static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) · line 2096

Quoted 7 lines · Yan Zhao
arch/x86/kvm/vmx/tdx.c +0 -0
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;

arch/x86/kvm/vmx/tdx.c +3 -0
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+

I don't think you should return RET_PF_RETRY here.

Quoted 2 lines

This is still at very early stage of EPT violation. The caller of tdx_handle_ept_violation() is expecting either 0, 1, or negative error code.

Hmm, strictly speaking, the caller of the EPT violation handler is expecting 0, >0, or negative error code.

vcpu_run

  |->r = vcpu_enter_guest(vcpu);
  |        |->r = kvm_x86_call(handle_exit)(vcpu, exit_fastpath);
  |        |  return r;
  |  if (r <= 0)
  |     break;

handle_ept_violation
|->return __vmx_handle_ept_violation(vcpu, gpa, exit_qualification);

tdx_handle_ept_violation
|->ret = __vmx_handle_ept_violation(vcpu, gpa, exit_qual); | return ret;

The current VMX/TDX's EPT violation handlers returns RET_PF_* to the caller since commit 7c5480386300 ("KVM: x86/mmu: Return RET_PF* instead of 1 in kvm_mmu_page_fault") for the sake of zero-step mitigation.

This is no problem, because

enum {
RET_PF_CONTINUE = 0,
RET_PF_RETRY,
RET_PF_EMULATE,
RET_PF_WRITE_PROTECTED,
RET_PF_INVALID,
RET_PF_FIXED,
RET_PF_SPURIOUS,
};

Quoted 7 lines

/*
* Define RET_PF_CONTINUE as 0 to allow for
* - efficient machine code when checking for CONTINUE, e.g.
* "TEST %rax, %rax, JNZ", as all "stop!" values are non-zero,
* - kvm_mmu_do_page_fault() to return other RET_PF_* as a positive value. */
static_assert(RET_PF_CONTINUE == 0);

Ah, OK.

But this makes KVM retry fault, when kvm_split_cross_boundary_leafs() fails, due to -ENOMEM, presumably. While in the normal page fault handler path, -ENOMEM will just return to userspace AFAICT.

This is not consistent, but I guess nobody cares, or noticed.

author
#83 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 56 lines · Kai
Quoted 47 lines · Yan Zhao
Quoted 11 lines · Kai

arch/x86/kvm/vmx/tdx.c static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) · line 2096

Quoted 7 lines · Yan Zhao
arch/x86/kvm/vmx/tdx.c +0 -0
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;

arch/x86/kvm/vmx/tdx.c +3 -0
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+

I don't think you should return RET_PF_RETRY here.

Quoted 2 lines

This is still at very early stage of EPT violation. The caller of tdx_handle_ept_violation() is expecting either 0, 1, or negative error code.

Hmm, strictly speaking, the caller of the EPT violation handler is expecting 0, >0, or negative error code.

vcpu_run

  |->r = vcpu_enter_guest(vcpu);
  |        |->r = kvm_x86_call(handle_exit)(vcpu, exit_fastpath);
  |        |  return r;
  |  if (r <= 0)
  |     break;

handle_ept_violation
|->return __vmx_handle_ept_violation(vcpu, gpa, exit_qualification);

tdx_handle_ept_violation
|->ret = __vmx_handle_ept_violation(vcpu, gpa, exit_qual); | return ret;

The current VMX/TDX's EPT violation handlers returns RET_PF_* to the caller since commit 7c5480386300 ("KVM: x86/mmu: Return RET_PF* instead of 1 in kvm_mmu_page_fault") for the sake of zero-step mitigation.

This is no problem, because

enum {
RET_PF_CONTINUE = 0,
RET_PF_RETRY,
RET_PF_EMULATE,
RET_PF_WRITE_PROTECTED,
RET_PF_INVALID,
RET_PF_FIXED,
RET_PF_SPURIOUS,
};

Quoted 7 lines

/*
* Define RET_PF_CONTINUE as 0 to allow for
* - efficient machine code when checking for CONTINUE, e.g.
* "TEST %rax, %rax, JNZ", as all "stop!" values are non-zero,
* - kvm_mmu_do_page_fault() to return other RET_PF_* as a positive value. */
static_assert(RET_PF_CONTINUE == 0);

Quoted 7 lines

Ah, OK.

But this makes KVM retry fault, when kvm_split_cross_boundary_leafs() fails, due to -ENOMEM, presumably. While in the normal page fault handler path, -ENOMEM will just return to userspace AFAICT.

This is not consistent, but I guess nobody cares, or noticed.

Oh, I got your point now.

Though retrying on -ENOMEM is also OK, returning ret to userspace for consistency is a good point, given mmu_topup_memory_caches() returns -ENOMEM to userspace.

I'll update it accordingly. Thanks!

#84 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

In reply to Yan Zhao
Quoted 93 lines · Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

The TDX module thus enables the EPT violation VMExit to carry the guest's accept level when the VMExit is caused by the guest's accept operation.

Therefore, in TDX's EPT violation handler
(1) Set the guest inhibit bit in the lpage info to prevent KVM MMU core from mapping at a higher a level than the guest's accept level.

(2) Split any existing huge mapping at the fault GFN to avoid unsupported splitting under the shared mmu_lock by TDX.

Use write mmu_lock to pretect (1) and (2) for now. If future KVM TDX can perform the actual splitting under shared mmu_lock with enhanced TDX modules, (1) is possible to be called under shared mmu_lock, and (2) would become unnecessary.

As an optimization, this patch calls hugepage_test_guest_inhibit() without holding the mmu_lock to reduce the frequency of acquiring the write mmu_lock. The write mmu_lock is thus only acquired if the guest inhibit bit is not already set. This is safe because the guest inhibit bit is set in a one-way manner while the splitting under the write mmu_lock is performed before setting the guest inhibit bit.

---
RFC v2
- Change tdx_get_accept_level() to tdx_check_accept_level().
- Invoke kvm_split_cross_boundary_leafs() and hugepage_set_guest_inhibit() to change KVM mapping level in a global way according to guest accept level. (Rick, Sean).

RFC v1:
- Introduce tdx_get_accept_level() to get guest accept level.
- Use tdx->violation_request_level and tdx->violation_gfn* to pass guest accept level to tdx_gmem_private_max_mapping_level() to detemine KVM mapping level.

---
 arch/x86/kvm/vmx/tdx.c      | 50 +++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx_arch.h |  3 +++
 2 files changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +34 -0
index 035d81275be4..71115058e5e6 100644
@@ -2019,6 +2019,53 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
 	return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
 }
 
+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(vcpu->kvm, gfn);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct kvm *kvm = vcpu->kvm;
+	u64 eeq_type, eeq_info;
+	int level = -1;
+
+	if (!slot)
+		return 0;
+
+	eeq_type = tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK;
+	if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_ACCEPT)
+		return 0;
+
+	eeq_info = (tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+		   TDX_EXT_EXIT_QUAL_INFO_SHIFT;
+
+	level = (eeq_info & GENMASK(2, 0)) + 1;
+
+	if (level == PG_LEVEL_4K || level == PG_LEVEL_2M) {
+		if (!hugepage_test_guest_inhibit(slot, gfn, level + 1)) {
+			gfn_t base_gfn = gfn_round_for_level(gfn, level);
+			struct kvm_gfn_range gfn_range = {
+				.start = base_gfn,
+				.end = base_gfn + KVM_PAGES_PER_HPAGE(level),
+				.slot = slot,
+				.may_block = true,
+				.attr_filter = KVM_FILTER_PRIVATE,
+			};
+
+			scoped_guard(write_lock, &kvm->mmu_lock) {
+				int ret;
+

arch/x86/kvm/vmx/tdx.c static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp · line 2065

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +10 -0
+				ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+				if (ret)
+					return ret;
+
+				hugepage_set_guest_inhibit(slot, gfn, level + 1);
+				if (level == PG_LEVEL_4K)
+					hugepage_set_guest_inhibit(slot, gfn, level + 2);
+			}
+		}
+	}

Also, could you also clarify what's the current behaviour when the exit doesn't have any level information?

Will 'level == PG_LEVEL_4K' in this case? Or will this function return early right after check the eeq_type?

It's not mentioned anywhere in the changelog. The cover letter vaguely says:

This mechanism allows support of huge pages for non-Linux TDs and also removes the 4KB restriction on pre-fault mappings for Linux TDs in RFC v1.

But it's not clear to me how this is solved.

author
#85 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 99 lines · Kai
Quoted 98 lines · Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

The TDX module thus enables the EPT violation VMExit to carry the guest's accept level when the VMExit is caused by the guest's accept operation.

Therefore, in TDX's EPT violation handler
(1) Set the guest inhibit bit in the lpage info to prevent KVM MMU core from mapping at a higher a level than the guest's accept level.

(2) Split any existing huge mapping at the fault GFN to avoid unsupported splitting under the shared mmu_lock by TDX.

Use write mmu_lock to pretect (1) and (2) for now. If future KVM TDX can perform the actual splitting under shared mmu_lock with enhanced TDX modules, (1) is possible to be called under shared mmu_lock, and (2) would become unnecessary.

As an optimization, this patch calls hugepage_test_guest_inhibit() without holding the mmu_lock to reduce the frequency of acquiring the write mmu_lock. The write mmu_lock is thus only acquired if the guest inhibit bit is not already set. This is safe because the guest inhibit bit is set in a one-way manner while the splitting under the write mmu_lock is performed before setting the guest inhibit bit.

---
RFC v2
- Change tdx_get_accept_level() to tdx_check_accept_level().
- Invoke kvm_split_cross_boundary_leafs() and hugepage_set_guest_inhibit() to change KVM mapping level in a global way according to guest accept level. (Rick, Sean).

RFC v1:
- Introduce tdx_get_accept_level() to get guest accept level.
- Use tdx->violation_request_level and tdx->violation_gfn* to pass guest accept level to tdx_gmem_private_max_mapping_level() to detemine KVM mapping level.

---
 arch/x86/kvm/vmx/tdx.c      | 50 +++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/vmx/tdx_arch.h |  3 +++
 2 files changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +39 -0
index 035d81275be4..71115058e5e6 100644
@@ -2019,6 +2019,53 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
 	return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
 }
 
+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(vcpu->kvm, gfn);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct kvm *kvm = vcpu->kvm;
+	u64 eeq_type, eeq_info;
+	int level = -1;
+
+	if (!slot)
+		return 0;
+
+	eeq_type = tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK;
+	if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_ACCEPT)
+		return 0;
+
+	eeq_info = (tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+		   TDX_EXT_EXIT_QUAL_INFO_SHIFT;
+
+	level = (eeq_info & GENMASK(2, 0)) + 1;
+
+	if (level == PG_LEVEL_4K || level == PG_LEVEL_2M) {
+		if (!hugepage_test_guest_inhibit(slot, gfn, level + 1)) {
+			gfn_t base_gfn = gfn_round_for_level(gfn, level);
+			struct kvm_gfn_range gfn_range = {
+				.start = base_gfn,
+				.end = base_gfn + KVM_PAGES_PER_HPAGE(level),
+				.slot = slot,
+				.may_block = true,
+				.attr_filter = KVM_FILTER_PRIVATE,
+			};
+
+			scoped_guard(write_lock, &kvm->mmu_lock) {
+				int ret;
+
+				ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+				if (ret)
+					return ret;
+
+				hugepage_set_guest_inhibit(slot, gfn, level + 1);
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp · line 2065

Quoted 5 lines
arch/x86/kvm/vmx/tdx.c +5 -0
+				if (level == PG_LEVEL_4K)
+					hugepage_set_guest_inhibit(slot, gfn, level + 2);
+			}
+		}
+	}

Also, could you also clarify what's the current behaviour when the exit doesn't have any level information?

An EPT violation exit seen by KVM for TDs is emulated by the TDX module. The TDX module provides VMM with more detailed info through the exit's extended exit qualification.

If an EPT violation exit is emulated due to the guest's ACCEPT operation, the extended exit qualification is of type TDX_EXT_EXIT_QUAL_TYPE_ACCEPT. Since an ACCEPT operation must provide a valid level (otherwise, the TDX module just fails guest ACCEPT without exit to VMM), the extended exit qualification info must carry a valid level too: either PG_LEVEL_4K or PG_LEVEL_2M.

So, if KVM sees an exit with no level info, the extended exit qualification is not of type TDX_EXT_EXIT_QUAL_TYPE_ACCEPT in the first place. It could be of type NONE or type PENDING_EPT_VIOLATION depending on whether the guest is configured with pending_ve_disable or if the gpa is private. This kind of exit is caused by guest accessing a memory without first accepting it.

Quoted 2 lines

Will 'level == PG_LEVEL_4K' in this case? Or will this function return early right after check the eeq_type?

The function will return early right after check the eeq_type.

Quoted 8 lines

It's not mentioned anywhere in the changelog. The cover letter vaguely says:

This mechanism allows support of huge pages for non-Linux TDs and also removes the 4KB restriction on pre-fault mappings for Linux TDs in RFC v1.

But it's not clear to me how this is solved.

I'll add a comment to tdx_check_accept_level() and update the patch log to make the picture clearer.

Thanks for pointing it out.

#86 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 3 lines · Yan Zhao
Quoted 2 lines

Will 'level == PG_LEVEL_4K' in this case?  Or will this function return early right after check the eeq_type?

The function will return early right after check the eeq_type.

But for such case the fault handler will still return 2M and KVM will AUG 2M page? Then if guest accepts 4K page, a new exit to KVM would happen?

But this time KVM is able to find the info that guest is accepting 4K and KVM will split the 2M to 4K pages so we are good to go?

author
#87 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 10 lines · Kai
Quoted 3 lines · Yan Zhao
Quoted 2 lines

Will 'level == PG_LEVEL_4K' in this case?  Or will this function return early right after check the eeq_type?

The function will return early right after check the eeq_type.

But for such case the fault handler will still return 2M and KVM will AUG 2M page? Then if guest accepts 4K page, a new exit to KVM would happen?

But this time KVM is able to find the info that guest is accepting 4K and KVM will split the 2M to 4K pages so we are good to go?

If guest accesses a private memory without first accepting it (like non-Linux guests), the sequence is:
1. Guest accesses a private memory.
2. KVM finds it can map the GFN at 2MB. So, AUG 2MB pages.
3. Guest accepts the GFN at 4KB.
4. KVM receives a EPT violation with eeq_type of ACCEPT and level 4KB
5. KVM splits the 2MB mapping.
6. Guest accepts successfully and accesses the page.

If guest first accepts a private memory before accessing (like Linux guests), the sequence is:
1. Guest accepts a private memory at 4KB.
2. KVM receives a EPT violation with eeq_type of ACCEPT and level 4KB.
3. KVM AUG 4KB.
4. Guest accepts successfully and accesses the page.

#88 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 12 lines · Yan Zhao
Quoted 10 lines · Kai
Quoted 3 lines · Yan Zhao
Quoted 2 lines

Will 'level == PG_LEVEL_4K' in this case?  Or will this function return early right after check the eeq_type?

The function will return early right after check the eeq_type.

But for such case the fault handler will still return 2M and KVM will AUG 2M page? Then if guest accepts 4K page, a new exit to KVM would happen?

But this time KVM is able to find the info that guest is accepting 4K and KVM will split the 2M to 4K pages so we are good to go?

Quoted 8 lines

If guest accesses a private memory without first accepting it (like non-Linux guests), the sequence is:
1. Guest accesses a private memory.
2. KVM finds it can map the GFN at 2MB. So, AUG 2MB pages.
3. Guest accepts the GFN at 4KB.
4. KVM receives a EPT violation with eeq_type of ACCEPT and level 4KB
5. KVM splits the 2MB mapping.
6. Guest accepts successfully and accesses the page.

Yeah looks good.

Btw, the change to make KVM AUG 2M when no accept level is specified is done in patch 23. I think you can add some text to explain in that patch?

E.g., something like:

Always try to AUG 2M hugepage, even there's no accept level from the guest. If the guest later accepts at 4K page, the TDX module will exit to KVM with the actual accept level info and KVM will split to 4K pages. The guest then will be able to accept the 4K pages successfully.

author
#89 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 27 lines · Kai
Quoted 12 lines · Yan Zhao
Quoted 10 lines · Kai
Quoted 3 lines · Yan Zhao
Quoted 2 lines

Will 'level == PG_LEVEL_4K' in this case?  Or will this function return early right after check the eeq_type?

The function will return early right after check the eeq_type.

But for such case the fault handler will still return 2M and KVM will AUG 2M page? Then if guest accepts 4K page, a new exit to KVM would happen?

But this time KVM is able to find the info that guest is accepting 4K and KVM will split the 2M to 4K pages so we are good to go?

Quoted 8 lines

If guest accesses a private memory without first accepting it (like non-Linux guests), the sequence is:
1. Guest accesses a private memory.
2. KVM finds it can map the GFN at 2MB. So, AUG 2MB pages.
3. Guest accepts the GFN at 4KB.
4. KVM receives a EPT violation with eeq_type of ACCEPT and level 4KB
5. KVM splits the 2MB mapping.
6. Guest accepts successfully and accesses the page.

Yeah looks good.

Btw, the change to make KVM AUG 2M when no accept level is specified is done in patch 23. I think you can add some text to explain in that patch?

Quoted 6 lines

E.g., something like:

Always try to AUG 2M hugepage, even there's no accept level from the guest. If the guest later accepts at 4K page, the TDX module will exit to KVM with the actual accept level info and KVM will split to 4K pages. The guest then will be able to accept the 4K pages successfully.

It's a good idea.
I think it's also better to mention in patch 23 that returning 2M in tdx_gmem_private_max_mapping_level() doesn't mean TDX will AUG 2M.

So, maybe
Always try to let KVM map at 2MB level, though KVM may still map the page at 4KB (i.e., passing in PG_LEVEL_4K to AUG) due to
- the backend folio is 4KB,
- disallow_lpage restrictions:
a) mixed private/shared pages in the 2MB range
b) level alignment due to slot base_gfn, slot size, and ugfn
c) guest_inhibit bit set due to guest accept level

When there's accept level of 4KB from the guest, KVM will AUG the page at 4KB directly due to the guest_inhibit bit set. So guest is able to accept at 4KB successfully.

When there's no accept level from the guest, and there're no other restrictions on the GFN range of a huge folio, KVM will AUG the page at 2MB first.

If the guest later accepts at 4K page, the TDX module will exit to KVM with the actual accept level info and KVM will split to 4K pages and set the guest_inhibit bit. The guest then will be able to accept the 4K pages successfully.

#90 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

In reply to Yan Zhao
Quoted 8 lines · Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

Quoted 6 lines

The TDX module thus enables the EPT violation VMExit to carry the guest's accept level when the VMExit is caused by the guest's accept operation.

Therefore, in TDX's EPT violation handler
(1) Set the guest inhibit bit in the lpage info to prevent KVM MMU core from mapping at a higher a level than the guest's accept level.

                             ^
                            an extra 'a'

Quoted 5 lines

(2) Split any existing huge mapping at the fault GFN to avoid unsupported splitting under the shared mmu_lock by TDX.

Use write mmu_lock to pretect (1) and (2) for now. If future KVM TDX can

pretect -> protect

Quoted 37 lines

perform the actual splitting under shared mmu_lock with enhanced TDX modules, (1) is possible to be called under shared mmu_lock, and (2) would become unnecessary.

As an optimization, this patch calls hugepage_test_guest_inhibit() without holding the mmu_lock to reduce the frequency of acquiring the write mmu_lock. The write mmu_lock is thus only acquired if the guest inhibit bit is not already set. This is safe because the guest inhibit bit is set in a one-way manner while the splitting under the write mmu_lock is performed before setting the guest inhibit bit.

---
RFC v2
- Change tdx_get_accept_level() to tdx_check_accept_level().
- Invoke kvm_split_cross_boundary_leafs() and hugepage_set_guest_inhibit() to change KVM mapping level in a global way according to guest accept level. (Rick, Sean).

RFC v1:
- Introduce tdx_get_accept_level() to get guest accept level.
- Use tdx->violation_request_level and tdx->violation_gfn* to pass guest accept level to tdx_gmem_private_max_mapping_level() to detemine KVM mapping level.

---
  arch/x86/kvm/vmx/tdx.c      | 50 +++++++++++++++++++++++++++++++++++++
  arch/x86/kvm/vmx/tdx_arch.h |  3 +++
  2 files changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +0 -0
index 035d81275be4..71115058e5e6 100644

arch/x86/kvm/vmx/tdx.c static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp · line 2022

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -2019,6 +2019,53 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
  	return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
  }
 
+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)

The function name sounds like it is just doing check, but it may split a hugepage on mismatch.

How about tdx_enforce_accept_level_mapping() or something else to reflect the change could be make?

Quoted 74 lines
arch/x86/kvm/vmx/tdx.c +49 -0
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(vcpu->kvm, gfn);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct kvm *kvm = vcpu->kvm;
+	u64 eeq_type, eeq_info;
+	int level = -1;
+
+	if (!slot)
+		return 0;
+
+	eeq_type = tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK;
+	if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_ACCEPT)
+		return 0;
+
+	eeq_info = (tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+		   TDX_EXT_EXIT_QUAL_INFO_SHIFT;
+
+	level = (eeq_info & GENMASK(2, 0)) + 1;
+
+	if (level == PG_LEVEL_4K || level == PG_LEVEL_2M) {
+		if (!hugepage_test_guest_inhibit(slot, gfn, level + 1)) {
+			gfn_t base_gfn = gfn_round_for_level(gfn, level);
+			struct kvm_gfn_range gfn_range = {
+				.start = base_gfn,
+				.end = base_gfn + KVM_PAGES_PER_HPAGE(level),
+				.slot = slot,
+				.may_block = true,
+				.attr_filter = KVM_FILTER_PRIVATE,
+			};
+
+			scoped_guard(write_lock, &kvm->mmu_lock) {
+				int ret;
+
+				ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+				if (ret)
+					return ret;
+
+				hugepage_set_guest_inhibit(slot, gfn, level + 1);
+				if (level == PG_LEVEL_4K)
+					hugepage_set_guest_inhibit(slot, gfn, level + 2);
+			}
+		}
+	}
+	return 0;
+}
+
  static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  {
  	unsigned long exit_qual;
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;
 
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+
  		/* Only private GPA triggers zero-step mitigation */
  		local_retry = true;
  	} else {
arch/x86/kvm/vmx/tdx_arch.h +3 -0
index a30e880849e3..af006a73ee05 100644
@@ -82,7 +82,10 @@ struct tdx_cpuid_value {
  #define TDX_TD_ATTR_PERFMON		BIT_ULL(63)
 
  #define TDX_EXT_EXIT_QUAL_TYPE_MASK	GENMASK(3, 0)
+#define TDX_EXT_EXIT_QUAL_TYPE_ACCEPT  1
  #define TDX_EXT_EXIT_QUAL_TYPE_PENDING_EPT_VIOLATION  6
+#define TDX_EXT_EXIT_QUAL_INFO_MASK	GENMASK(63, 32)
+#define TDX_EXT_EXIT_QUAL_INFO_SHIFT	32
  /*
   * TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B.
   */
author
#91 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

Quoted 11 lines · Binbin Wu
Quoted 8 lines · Yan Zhao

TDX requires guests to accept S-EPT mappings created by the host KVM. Due to the current implementation of the TDX module, if a guest accepts a GFN at a lower level after KVM maps it at a higher level, the TDX module will emulate an EPT violation VMExit to KVM instead of returning a size mismatch error to the guest. If KVM fails to perform page splitting in the VMExit handler, the guest's accept operation will be triggered again upon re-entering the guest, causing a repeated EPT violation VMExit.

Quoted 8 lines
Quoted 6 lines

The TDX module thus enables the EPT violation VMExit to carry the guest's accept level when the VMExit is caused by the guest's accept operation.

Therefore, in TDX's EPT violation handler
(1) Set the guest inhibit bit in the lpage info to prevent KVM MMU core from mapping at a higher a level than the guest's accept level.

                             ^
                            an extra 'a'

Thanks.

Quoted 7 lines
Quoted 5 lines

(2) Split any existing huge mapping at the fault GFN to avoid unsupported splitting under the shared mmu_lock by TDX.

Use write mmu_lock to pretect (1) and (2) for now. If future KVM TDX can

pretect -> protect

Thanks.

Quoted 44 lines
Quoted 37 lines

perform the actual splitting under shared mmu_lock with enhanced TDX modules, (1) is possible to be called under shared mmu_lock, and (2) would become unnecessary.

As an optimization, this patch calls hugepage_test_guest_inhibit() without holding the mmu_lock to reduce the frequency of acquiring the write mmu_lock. The write mmu_lock is thus only acquired if the guest inhibit bit is not already set. This is safe because the guest inhibit bit is set in a one-way manner while the splitting under the write mmu_lock is performed before setting the guest inhibit bit.

---
RFC v2
- Change tdx_get_accept_level() to tdx_check_accept_level().
- Invoke kvm_split_cross_boundary_leafs() and hugepage_set_guest_inhibit() to change KVM mapping level in a global way according to guest accept level. (Rick, Sean).

RFC v1:
- Introduce tdx_get_accept_level() to get guest accept level.
- Use tdx->violation_request_level and tdx->violation_gfn* to pass guest accept level to tdx_gmem_private_max_mapping_level() to detemine KVM mapping level.

---
  arch/x86/kvm/vmx/tdx.c      | 50 +++++++++++++++++++++++++++++++++++++
  arch/x86/kvm/vmx/tdx_arch.h |  3 +++
  2 files changed, 53 insertions(+)
arch/x86/kvm/vmx/tdx.c +0 -0
index 035d81275be4..71115058e5e6 100644

arch/x86/kvm/vmx/tdx.c static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp · line 2021

Quoted 3 lines
arch/x86/kvm/vmx/tdx.c +1 -0
@@ -2019,6 +2019,53 @@ static inline bool tdx_is_sept_violation_unexpected_pending(struct kvm_vcpu *vcp
  	return !(eq & EPT_VIOLATION_PROT_MASK) && !(eq & EPT_VIOLATION_EXEC_FOR_RING3_LIN);
  }
+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)

The function name sounds like it is just doing check, but it may split a hugepage on mismatch.

Quoted 2 lines

How about tdx_enforce_accept_level_mapping() or something else to reflect the change could be make?

What about tdx_honor_guest_accept_level()?

Quoted 73 lines
Quoted 72 lines
arch/x86/kvm/vmx/tdx.c +49 -0
+{
+	struct kvm_memory_slot *slot = gfn_to_memslot(vcpu->kvm, gfn);
+	struct vcpu_tdx *tdx = to_tdx(vcpu);
+	struct kvm *kvm = vcpu->kvm;
+	u64 eeq_type, eeq_info;
+	int level = -1;
+
+	if (!slot)
+		return 0;
+
+	eeq_type = tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_TYPE_MASK;
+	if (eeq_type != TDX_EXT_EXIT_QUAL_TYPE_ACCEPT)
+		return 0;
+
+	eeq_info = (tdx->ext_exit_qualification & TDX_EXT_EXIT_QUAL_INFO_MASK) >>
+		   TDX_EXT_EXIT_QUAL_INFO_SHIFT;
+
+	level = (eeq_info & GENMASK(2, 0)) + 1;
+
+	if (level == PG_LEVEL_4K || level == PG_LEVEL_2M) {
+		if (!hugepage_test_guest_inhibit(slot, gfn, level + 1)) {
+			gfn_t base_gfn = gfn_round_for_level(gfn, level);
+			struct kvm_gfn_range gfn_range = {
+				.start = base_gfn,
+				.end = base_gfn + KVM_PAGES_PER_HPAGE(level),
+				.slot = slot,
+				.may_block = true,
+				.attr_filter = KVM_FILTER_PRIVATE,
+			};
+
+			scoped_guard(write_lock, &kvm->mmu_lock) {
+				int ret;
+
+				ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+				if (ret)
+					return ret;
+
+				hugepage_set_guest_inhibit(slot, gfn, level + 1);
+				if (level == PG_LEVEL_4K)
+					hugepage_set_guest_inhibit(slot, gfn, level + 2);
+			}
+		}
+	}
+	return 0;
+}
+
  static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  {
  	unsigned long exit_qual;
@@ -2044,6 +2091,9 @@ static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu)
  		 */
  		exit_qual = EPT_VIOLATION_ACC_WRITE;
+		if (tdx_check_accept_level(vcpu, gpa_to_gfn(gpa)))
+			return RET_PF_RETRY;
+
  		/* Only private GPA triggers zero-step mitigation */
  		local_retry = true;
  	} else {
arch/x86/kvm/vmx/tdx_arch.h +3 -0
index a30e880849e3..af006a73ee05 100644
@@ -82,7 +82,10 @@ struct tdx_cpuid_value {
  #define TDX_TD_ATTR_PERFMON		BIT_ULL(63)
  #define TDX_EXT_EXIT_QUAL_TYPE_MASK	GENMASK(3, 0)
+#define TDX_EXT_EXIT_QUAL_TYPE_ACCEPT  1
  #define TDX_EXT_EXIT_QUAL_TYPE_PENDING_EPT_VIOLATION  6
+#define TDX_EXT_EXIT_QUAL_INFO_MASK	GENMASK(63, 32)
+#define TDX_EXT_EXIT_QUAL_INFO_SHIFT	32
  /*
   * TD_PARAMS is provided as an input to TDH_MNG_INIT, the size of which is 1024B.
   */
#92 lore raw

Re: [RFC PATCH v2 14/23] KVM: TDX: Split and inhibit huge mappings if a VMExit carries level info

On 11/19/2025 2:29 PM, Yan Zhao wrote:
[...]

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

+static inline int tdx_check_accept_level(struct kvm_vcpu *vcpu, gfn_t gfn)

The function name sounds like it is just doing check, but it may split a hugepage on mismatch.

How about tdx_enforce_accept_level_mapping() or something else to reflect the change could be make?

What about tdx_honor_guest_accept_level()?

It looks good to me.

author
#93 lore raw

RFC PATCH v2 15/23 KVM: Change the return type of gfn_handler_t() from bool to int

In reply to Yan Zhao

Modify the return type of gfn_handler_t() from bool to int. A negative return value indicates failure, while a return value of 1 signifies success with a flush required, and 0 denotes success without a flush required.

This adjustment prepares for a later change that will enable kvm_pre_set_memory_attributes() to fail.

No functional changes expected.

---
RFC v2:
- No change

RFC v1:
- New patch.

---
 arch/arm64/kvm/mmu.c             |  8 ++++----
 arch/loongarch/kvm/mmu.c         |  8 ++++----
 arch/mips/kvm/mmu.c              |  6 +++---
 arch/powerpc/kvm/book3s.c        |  4 ++--
 arch/powerpc/kvm/e500_mmu_host.c |  8 ++++----
 arch/riscv/kvm/mmu.c             | 12 ++++++------
 arch/x86/kvm/mmu/mmu.c           | 20 ++++++++++----------
 include/linux/kvm_host.h         | 12 ++++++------
 virt/kvm/kvm_main.c              | 24 ++++++++++++++++--------
 9 files changed, 55 insertions(+), 47 deletions(-)
arch/arm64/kvm/mmu.c +4 -4
index 8b225450a4eb..991a6df0ca21 100644
@@ -1999,12 +1999,12 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	return false;
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	u64 size = (range->end - range->start) << PAGE_SHIFT;
 
 	if (!kvm->arch.mmu.pgt)
-		return false;
+		return 0;
 
 	return KVM_PGT_FN(kvm_pgtable_stage2_test_clear_young)(kvm->arch.mmu.pgt,
 						   range->start << PAGE_SHIFT,
@@ -2015,12 +2015,12 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 	 */
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	u64 size = (range->end - range->start) << PAGE_SHIFT;
 
 	if (!kvm->arch.mmu.pgt)
-		return false;
+		return 0;
 
 	return KVM_PGT_FN(kvm_pgtable_stage2_test_clear_young)(kvm->arch.mmu.pgt,
 						   range->start << PAGE_SHIFT,
arch/loongarch/kvm/mmu.c +4 -4
index ed956c5cf2cc..0542516c98eb 100644
@@ -511,7 +511,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 			range->end << PAGE_SHIFT, &ctx);
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_ptw_ctx ctx;
 
@@ -523,15 +523,15 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 				range->end << PAGE_SHIFT, &ctx);
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	gpa_t gpa = range->start << PAGE_SHIFT;
 	kvm_pte_t *ptep = kvm_populate_gpa(kvm, NULL, gpa, 0);
 
 	if (ptep && kvm_pte_present(NULL, ptep) && kvm_pte_young(*ptep))
-		return true;
+		return 1;
 
-	return false;
+	return 0;
 }
 
 /*
arch/mips/kvm/mmu.c +3 -3
index d2c3b6b41f18..c26cc89c8e98 100644
@@ -444,18 +444,18 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	return true;
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	return kvm_mips_mkold_gpa_pt(kvm, range->start, range->end);
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	gpa_t gpa = range->start << PAGE_SHIFT;
 	pte_t *gpa_pte = kvm_mips_pte_for_gpa(kvm, NULL, gpa);
 
 	if (!gpa_pte)
-		return false;
+		return 0;
 	return pte_young(*gpa_pte);
 }
 
arch/powerpc/kvm/book3s.c +2 -2
index d79c5d1098c0..9bf6e1cf64f1 100644
@@ -886,12 +886,12 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	return kvm->arch.kvm_ops->unmap_gfn_range(kvm, range);
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	return kvm->arch.kvm_ops->age_gfn(kvm, range);
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	return kvm->arch.kvm_ops->test_age_gfn(kvm, range);
 }
arch/powerpc/kvm/e500_mmu_host.c +4 -4
index 06caf8bbbe2b..dd5411ee242e 100644
@@ -697,16 +697,16 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	return kvm_e500_mmu_unmap_gfn(kvm, range);
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	/* XXX could be more clever ;) */
-	return false;
+	return 0;
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	/* XXX could be more clever ;) */
-	return false;
+	return 0;
 }
 
 /*****************************************/
arch/riscv/kvm/mmu.c +6 -6
index 1087ea74567b..98c2fcd9229f 100644
@@ -550,38 +550,38 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 	return false;
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	pte_t *ptep;
 	u32 ptep_level = 0;
 	u64 size = (range->end - range->start) << PAGE_SHIFT;
 
 	if (!kvm->arch.pgd)
-		return false;
+		return 0;
 
 	WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
 
 	if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,
 				   &ptep, &ptep_level))
-		return false;
+		return 0;
 
 	return ptep_test_and_clear_young(NULL, 0, ptep);
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	pte_t *ptep;
 	u32 ptep_level = 0;
 	u64 size = (range->end - range->start) << PAGE_SHIFT;
 
 	if (!kvm->arch.pgd)
-		return false;
+		return 0;
 
 	WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE);
 
 	if (!gstage_get_leaf_entry(kvm, range->start << PAGE_SHIFT,
 				   &ptep, &ptep_level))
-		return false;
+		return 0;
 
 	return pte_young(ptep_get(ptep));
 }
arch/x86/kvm/mmu/mmu.c +10 -10
index 1c639286aac2..c71f8bb0b903 100644
@@ -1806,7 +1806,7 @@ static bool kvm_may_have_shadow_mmu_sptes(struct kvm *kvm)
 	return !tdp_mmu_enabled || READ_ONCE(kvm->arch.indirect_shadow_pages);
 }
 
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
@@ -1819,7 +1819,7 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 	return young;
 }
 
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	bool young = false;
 
@@ -7841,8 +7841,8 @@ static void hugepage_set_mixed(struct kvm_memory_slot *slot, gfn_t gfn,
 	lpage_info_slot(gfn, slot, level)->disallow_lpage |= KVM_LPAGE_MIXED_FLAG;
 }
 
-bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
-					struct kvm_gfn_range *range)
+int kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
+				       struct kvm_gfn_range *range)
 {
 	struct kvm_memory_slot *slot = range->slot;
 	int level;
@@ -7859,10 +7859,10 @@ bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 	 * a hugepage can be used for affected ranges.
 	 */
 	if (WARN_ON_ONCE(!kvm_arch_supports_gmem(kvm)))
-		return false;
+		return 0;
 
 	if (WARN_ON_ONCE(range->end <= range->start))
-		return false;
+		return 0;
 
 	/*
 	 * If the head and tail pages of the range currently allow a hugepage,
@@ -7921,8 +7921,8 @@ static bool hugepage_has_attrs(struct kvm *kvm, struct kvm_memory_slot *slot,
 	return true;
 }
 
-bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
-					 struct kvm_gfn_range *range)
+int kvm_arch_post_set_memory_attributes(struct kvm *kvm,
+					struct kvm_gfn_range *range)
 {
 	unsigned long attrs = range->arg.attributes;
 	struct kvm_memory_slot *slot = range->slot;
@@ -7938,7 +7938,7 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 	 * SHARED may now allow hugepages.
 	 */
 	if (WARN_ON_ONCE(!kvm_arch_supports_gmem(kvm)))
-		return false;
+		return 0;
 
 	/*
 	 * The sequence matters here: upper levels consume the result of lower
@@ -7985,7 +7985,7 @@ bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 				hugepage_set_mixed(slot, gfn, level);
 		}
 	}
-	return false;
+	return 0;
 }
 
 void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm,
include/linux/kvm_host.h +6 -6
index 6137b76341e1..d03e4a70a6db 100644
@@ -271,8 +271,8 @@ struct kvm_gfn_range {
 	bool lockless;
 };
 bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
-bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
-bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
+int kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
+int kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range);
 int kvm_split_cross_boundary_leafs(struct kvm *kvm, struct kvm_gfn_range *range,
 				   bool shared);
 #endif
@@ -1537,7 +1537,7 @@ void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
 void kvm_mmu_invalidate_begin(struct kvm *kvm);
 void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end);
 void kvm_mmu_invalidate_end(struct kvm *kvm);
-bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
+int kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range);
 
 long kvm_arch_dev_ioctl(struct file *filp,
 			unsigned int ioctl, unsigned long arg);
@@ -2524,10 +2524,10 @@ static inline unsigned long kvm_get_memory_attributes(struct kvm *kvm, gfn_t gfn
 
 bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 				     unsigned long mask, unsigned long attrs);
-bool kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
+int kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
+				       struct kvm_gfn_range *range);
+int kvm_arch_post_set_memory_attributes(struct kvm *kvm,
 					struct kvm_gfn_range *range);
-bool kvm_arch_post_set_memory_attributes(struct kvm *kvm,
-					 struct kvm_gfn_range *range);
 
 /*
  * Returns true if the given gfn's private/shared status (in the CoCo sense) is
virt/kvm/kvm_main.c +16 -8
index fe86f3f627ba..8f87d6c6be3f 100644
@@ -508,7 +508,7 @@ static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn)
 	return container_of(mn, struct kvm, mmu_notifier);
 }
 
-typedef bool (*gfn_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range);
+typedef int (*gfn_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range);
 
 typedef void (*on_lock_fn_t)(struct kvm *kvm);
 
@@ -592,6 +592,7 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 		kvm_for_each_memslot_in_hva_range(node, slots,
 						  range->start, range->end - 1) {
 			unsigned long hva_start, hva_end;
+			int ret;
 
 			slot = container_of(node, struct kvm_memory_slot, hva_node[slots->node_idx]);
 			hva_start = max_t(unsigned long, range->start, slot->userspace_addr);
@@ -632,7 +633,9 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm,
 						goto mmu_unlock;
 				}
 			}
-			r.ret |= range->handler(kvm, &gfn_range);
+			ret = range->handler(kvm, &gfn_range);
+			WARN_ON_ONCE(ret < 0);
+			r.ret |= ret;
 		}
 	}
 
@@ -718,7 +721,7 @@ void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end)
 	}
 }
 
-bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
+int kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_mmu_invalidate_range_add(kvm, range->start, range->end);
 	return kvm_unmap_gfn_range(kvm, range);
@@ -2469,7 +2472,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 	struct kvm_memslots *slots;
 	struct kvm_memslot_iter iter;
 	bool found_memslot = false;
-	bool ret = false;
+	bool flush = false;
+	int ret = 0;
 	int i;
 
 	gfn_range.arg = range->arg;
@@ -2502,19 +2506,23 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 					range->on_lock(kvm);
 			}
 
-			ret |= range->handler(kvm, &gfn_range);
+			ret = range->handler(kvm, &gfn_range);
+			if (ret < 0)
+				goto err;
+			flush |= ret;
 		}
 	}
 
-	if (range->flush_on_ret && ret)
+err:
+	if (range->flush_on_ret && flush)
 		kvm_flush_remote_tlbs(kvm);
 
 	if (found_memslot)
 		KVM_MMU_UNLOCK(kvm);
 }
 
-static bool kvm_pre_set_memory_attributes(struct kvm *kvm,
-					  struct kvm_gfn_range *range)
+static int kvm_pre_set_memory_attributes(struct kvm *kvm,
+					 struct kvm_gfn_range *range)
 {
 	/*
 	 * Unconditionally add the range to the invalidation set, regardless of
Signature · 1 line
2.43.2
author
#94 lore raw

RFC PATCH v2 16/23 KVM: x86: Split cross-boundary mirror leafs for KVM_SET_MEMORY_ATTRIBUTES

In reply to Yan Zhao

In TDX, private page tables require precise zapping because faulting back the zapped mappings necessitates the guest's re-acceptance. Therefore, before performing a zap for the private-to-shared conversion, rather than zapping a huge leaf entry that crosses the boundary of the GFN range to be zapped, split the leaf entry to ensure GFNs outside the conversion range are not affected.

Invoke kvm_split_cross_boundary_leafs() in kvm_arch_pre_set_memory_attributes() to split the huge leafs that cross GFN range boundary before calling kvm_unmap_gfn_range() to zap the GFN range that will be converted to shared.
When kvm_split_cross_boundary_leafs() fails, it is expected to internally invoke kvm_flush_remote_tlbs() to flush any changes that have been successfully completed.

Unlike kvm_unmap_gfn_range(), which cannot fail, kvm_split_cross_boundary_leafs() may fail due to memory allocation for splitting. Update kvm_handle_gfn_range() to propagate the error back to kvm_vm_set_mem_attributes(), which can then fail the ioctl KVM_SET_MEMORY_ATTRIBUTES.

The downside of current implementation is that though kvm_split_cross_boundary_leafs() is invoked before kvm_unmap_gfn_range() for each GFN range, the entire conversion range may consist of several GFN ranges. If an out-of-memory error occurs during the splitting of a GFN range, some previous GFN ranges may have been successfully split and zapped, even though their page attributes remain unchanged due to the splitting failure.

If it's necessary, a patch can be arranged to divide a single invocation of "kvm_handle_gfn_range(kvm, &pre_set_range)" into two, e.g.,

kvm_handle_gfn_range(kvm, &pre_set_range_prepare_and_split) kvm_handle_gfn_range(kvm, &pre_set_range_unmap),

---
RFC v2:
- update kvm_split_boundary_leafs() to kvm_split_cross_boundary_leafs() and invoke it only for priate-to-shared conversion.

RFC v1:
- new patch.

---
 arch/x86/kvm/mmu/mmu.c | 13 ++++++++++---
 virt/kvm/kvm_main.c    | 13 +++++++++----
 2 files changed, 19 insertions(+), 7 deletions(-)
arch/x86/kvm/mmu/mmu.c +10 -3
index c71f8bb0b903..f23d8fc59323 100644
@@ -7845,7 +7845,9 @@ int kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 				       struct kvm_gfn_range *range)
 {
 	struct kvm_memory_slot *slot = range->slot;
+	bool flush = false;
 	int level;
+	int ret;
 
 	/*
 	 * Zap SPTEs even if the slot can't be mapped PRIVATE.  KVM x86 only
@@ -7894,12 +7896,17 @@ int kvm_arch_pre_set_memory_attributes(struct kvm *kvm,
 	}
 
 	/* Unmap the old attribute page. */
-	if (range->arg.attributes & KVM_MEMORY_ATTRIBUTE_PRIVATE)
+	if (range->arg.attributes & KVM_MEMORY_ATTRIBUTE_PRIVATE) {
 		range->attr_filter = KVM_FILTER_SHARED;
-	else
+	} else {
 		range->attr_filter = KVM_FILTER_PRIVATE;
+		ret = kvm_split_cross_boundary_leafs(kvm, range, false);
+		if (ret < 0)
+			return ret;
+		flush |= ret;
+	}
 
-	return kvm_unmap_gfn_range(kvm, range);
+	return kvm_unmap_gfn_range(kvm, range) | flush;
 }
 
 
virt/kvm/kvm_main.c +9 -4
index 8f87d6c6be3f..9dceecf34822 100644
@@ -2464,8 +2464,8 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 	return true;
 }
 
-static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
-						 struct kvm_mmu_notifier_range *range)
+static __always_inline int kvm_handle_gfn_range(struct kvm *kvm,
+						struct kvm_mmu_notifier_range *range)
 {
 	struct kvm_gfn_range gfn_range;
 	struct kvm_memory_slot *slot;
@@ -2519,6 +2519,8 @@ static __always_inline void kvm_handle_gfn_range(struct kvm *kvm,
 
 	if (found_memslot)
 		KVM_MMU_UNLOCK(kvm);
+
+	return ret < 0 ? ret : 0;
 }
 
 static int kvm_pre_set_memory_attributes(struct kvm *kvm,
@@ -2587,7 +2589,9 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		cond_resched();
 	}
 
-	kvm_handle_gfn_range(kvm, &pre_set_range);
+	r = kvm_handle_gfn_range(kvm, &pre_set_range);
+	if (r)
+		goto out_unlock;
 
 	for (i = start; i < end; i++) {
 		r = xa_err(xa_store(&kvm->mem_attr_array, i, entry,
@@ -2596,7 +2600,8 @@ static int kvm_vm_set_mem_attributes(struct kvm *kvm, gfn_t start, gfn_t end,
 		cond_resched();
 	}
 
-	kvm_handle_gfn_range(kvm, &post_set_range);
+	r = kvm_handle_gfn_range(kvm, &post_set_range);
+	KVM_BUG_ON(r, kvm);
 
 out_unlock:
 	mutex_unlock(&kvm->slots_lock);
Signature · 1 line
2.43.2
author
#95 lore raw

RFC PATCH v2 17/23 KVM: guest_memfd: Split for punch hole and private-to-shared conversion

In reply to Yan Zhao

In TDX, private page tables require precise zapping because faulting back the zapped mappings necessitates the guest's re-acceptance. Therefore, before performing a zap for hole punching and private-to-shared conversions, huge leafs that cross the boundary of the zapping GFN range in the mirror page table must be split.

Splitting may result in an error. If this happens, hole punching and private-to-shared conversion should bail out early and return an error to userspace.

Splitting is not necessary for kvm_gmem_release() since the entire page table is being zapped, nor for kvm_gmem_error_folio() as an SPTE must not map more than one physical folio.

Therefore, in this patch,
- break kvm_gmem_invalidate_begin_and_zap() into kvm_gmem_invalidate_begin() and kvm_gmem_zap() and have kvm_gmem_release() and kvm_gmem_error_folio() to invoke them.

- have kvm_gmem_punch_hole() to invoke kvm_gmem_invalidate_begin(), kvm_gmem_split_private(), and kvm_gmem_zap().
Bail out if kvm_gmem_split_private() returns error.

- drop the old kvm_gmem_unmap_private() and have private-to-shared conversion to invoke kvm_gmem_split_private() and kvm_gmem_zap() instead. Bail out if kvm_gmem_split_private() returns error.

---
RFC v2:
- Rebased to [1]. As changes in this patch are gmem specific, they may need to be updated if the implementation in [1] changes.
- Update kvm_split_boundary_leafs() to kvm_split_cross_boundary_leafs() and invoke it before kvm_gmem_punch_hole() and private-to-shared conversion.

[1] https://lore.kernel.org/all/[email protected]/

RFC v1:
- new patch.

---
 virt/kvm/guest_memfd.c | 142 ++++++++++++++++++++++++-----------------
 1 file changed, 84 insertions(+), 58 deletions(-)
virt/kvm/guest_memfd.c +84 -58
index 67aa2285aa49..9edf33c482d7 100644
@@ -318,14 +318,14 @@ static bool kvm_gmem_has_safe_refcount(struct address_space *mapping, pgoff_t st
 	return refcount_safe;
 }
 
-static void kvm_gmem_unmap_private(struct kvm_gmem *gmem, pgoff_t start,
-				   pgoff_t end)
+static int kvm_gmem_split_private(struct kvm_gmem *gmem, pgoff_t start, pgoff_t end)
 {
 	struct kvm_memory_slot *slot;
 	struct kvm *kvm = gmem->kvm;
 	unsigned long index;
 	bool locked = false;
 	bool flush = false;
+	int ret = 0;
 
 	xa_for_each_range(&gmem->bindings, index, slot, start, end - 1) {
 		pgoff_t pgoff = slot->gmem.pgoff;
@@ -335,7 +335,6 @@ static void kvm_gmem_unmap_private(struct kvm_gmem *gmem, pgoff_t start,
 			.end = slot->base_gfn + min(pgoff + slot->npages, end) - pgoff,
 			.slot = slot,
 			.may_block = true,
-			/* This function is only concerned with private mappings. */
 			.attr_filter = KVM_FILTER_PRIVATE,
 		};
 
@@ -344,6 +343,47 @@ static void kvm_gmem_unmap_private(struct kvm_gmem *gmem, pgoff_t start,
 			locked = true;
 		}
 
+		ret = kvm_split_cross_boundary_leafs(kvm, &gfn_range, false);
+		if (ret < 0)
+			goto out;
+
+		flush |= ret;
+		ret = 0;
+	}
+out:
+	if (flush)
+		kvm_flush_remote_tlbs(kvm);
+
+	if (locked)
+		KVM_MMU_UNLOCK(kvm);
+
+	return ret;
+}
+
+static void kvm_gmem_zap(struct kvm_gmem *gmem, pgoff_t start, pgoff_t end,
+			 enum kvm_gfn_range_filter filter)
+{
+	struct kvm_memory_slot *slot;
+	struct kvm *kvm = gmem->kvm;
+	unsigned long index;
+	bool locked = false;
+	bool flush = false;
+
+	xa_for_each_range(&gmem->bindings, index, slot, start, end - 1) {
+		pgoff_t pgoff = slot->gmem.pgoff;
+		struct kvm_gfn_range gfn_range = {
+			.start = slot->base_gfn + max(pgoff, start) - pgoff,
+			.end = slot->base_gfn + min(pgoff + slot->npages, end) - pgoff,
+			.slot = slot,
+			.may_block = true,
+			.attr_filter = filter,
+		};
+
+		if (!locked) {
+			KVM_MMU_LOCK(kvm);
+			locked = true;
+		}
+
 		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
 	}
 
@@ -514,6 +554,8 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
 					   struct conversion_work *work,
 					   bool to_shared, pgoff_t *error_index)
 {
+	int ret = 0;
+
 	if (to_shared) {
 		struct list_head *gmem_list;
 		struct kvm_gmem *gmem;
@@ -522,19 +564,24 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
 		work_end = work->start + work->nr_pages;
 
 		gmem_list = &inode->i_mapping->i_private_list;
+		list_for_each_entry(gmem, gmem_list, entry) {
+			ret = kvm_gmem_split_private(gmem, work->start, work_end);
+			if (ret)
+				return ret;
+		}
 		list_for_each_entry(gmem, gmem_list, entry)
-			kvm_gmem_unmap_private(gmem, work->start, work_end);
+			kvm_gmem_zap(gmem, work->start, work_end, KVM_FILTER_PRIVATE);
 	} else {
 		unmap_mapping_pages(inode->i_mapping, work->start,
 				    work->nr_pages, false);
 
 		if (!kvm_gmem_has_safe_refcount(inode->i_mapping, work->start,
 						work->nr_pages, error_index)) {
-			return -EAGAIN;
+			ret = -EAGAIN;
 		}
 	}
 
-	return 0;
+	return ret;
 }
 
 static int kvm_gmem_restructure_folios_in_range(struct inode *inode,
@@ -1187,54 +1234,6 @@ static struct folio *kvm_gmem_get_folio(struct inode *inode, pgoff_t index)
 	return ERR_PTR(ret);
 }
 
-static void kvm_gmem_invalidate_begin_and_zap(struct kvm_gmem *gmem,
-					      pgoff_t start, pgoff_t end)
-{
-	bool flush = false, found_memslot = false;
-	struct kvm_memory_slot *slot;
-	struct kvm *kvm = gmem->kvm;
-	unsigned long index;
-
-	xa_for_each_range(&gmem->bindings, index, slot, start, end - 1) {
-		enum kvm_gfn_range_filter filter;
-		pgoff_t pgoff = slot->gmem.pgoff;
-
-		filter = KVM_FILTER_PRIVATE;
-		if (kvm_gmem_memslot_supports_shared(slot)) {
-			/*
-			 * Unmapping would also cause invalidation, but cannot
-			 * rely on mmu_notifiers to do invalidation via
-			 * unmapping, since memory may not be mapped to
-			 * userspace.
-			 */
-			filter |= KVM_FILTER_SHARED;
-		}
-
-		struct kvm_gfn_range gfn_range = {
-			.start = slot->base_gfn + max(pgoff, start) - pgoff,
-			.end = slot->base_gfn + min(pgoff + slot->npages, end) - pgoff,
-			.slot = slot,
-			.may_block = true,
-			.attr_filter = filter,
-		};
-
-		if (!found_memslot) {
-			found_memslot = true;
-
-			KVM_MMU_LOCK(kvm);
-			kvm_mmu_invalidate_begin(kvm);
-		}
-
-		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
-	}
-
-	if (flush)
-		kvm_flush_remote_tlbs(kvm);
-
-	if (found_memslot)
-		KVM_MMU_UNLOCK(kvm);
-}
-
 static void kvm_gmem_invalidate_end(struct kvm_gmem *gmem, pgoff_t start,
 				    pgoff_t end)
 {
@@ -1445,9 +1444,28 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
 	filemap_invalidate_lock(inode->i_mapping);
 
 	list_for_each_entry(gmem, gmem_list, entry)
-		kvm_gmem_invalidate_begin_and_zap(gmem, start, end);
+		kvm_gmem_invalidate_begin(gmem, start, end);
 
 	ret = 0;
+	list_for_each_entry(gmem, gmem_list, entry) {
+		ret = kvm_gmem_split_private(gmem, start, end);
+		if (ret)
+			goto out;
+	}
+	list_for_each_entry(gmem, gmem_list, entry) {
+		enum kvm_gfn_range_filter filter;
+
+		/*
+		 * kvm_gmem_invalidate_begin() would have unmapped shared
+		 * mappings via mmu notifiers, but only if those mappings were
+		 * actually set up. Since guest_memfd cannot assume that shared
+		 * mappings were set up, zap both private and shared mappings
+		 * here. If shared mappings were zapped, this should not be
+		 * expensive.
+		 */
+		filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED;
+		kvm_gmem_zap(gmem, start, end, filter);
+	}
 	if (kvm_gmem_has_custom_allocator(inode)) {
 		ret = kvm_gmem_truncate_inode_range(inode, offset, offset + len);
 	} else {
@@ -1455,6 +1473,7 @@ static long kvm_gmem_punch_hole(struct inode *inode, loff_t offset, loff_t len)
 		truncate_inode_pages_range(inode->i_mapping, offset, offset + len - 1);
 	}
 
+out:
 	list_for_each_entry(gmem, gmem_list, entry)
 		kvm_gmem_invalidate_end(gmem, start, end);
 
@@ -1576,7 +1595,8 @@ static int kvm_gmem_release(struct inode *inode, struct file *file)
 	 * Zap all SPTEs pointed at by this file.  Do not free the backing
 	 * memory, as its lifetime is associated with the inode, not the file.
 	 */
-	kvm_gmem_invalidate_begin_and_zap(gmem, 0, -1ul);
+	kvm_gmem_invalidate_begin(gmem, 0, -1ul);
+	kvm_gmem_zap(gmem, 0, -1ul, KVM_FILTER_PRIVATE | KVM_FILTER_SHARED);
 	kvm_gmem_invalidate_end(gmem, 0, -1ul);
 
 	list_del(&gmem->entry);
@@ -1906,8 +1926,14 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	start = folio->index;
 	end = start + folio_nr_pages(folio);
 
-	list_for_each_entry(gmem, gmem_list, entry)
-		kvm_gmem_invalidate_begin_and_zap(gmem, start, end);
+	/* The size of the SEPT will not exceed the size of the folio */
+	list_for_each_entry(gmem, gmem_list, entry) {
+		enum kvm_gfn_range_filter filter;
+
+		kvm_gmem_invalidate_begin(gmem, start, end);
+		filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED;
+		kvm_gmem_zap(gmem, start, end, filter);
+	}
 
 	/*
 	 * Do not truncate the range, what action is taken in response to the
Signature · 1 line
2.43.2
#96 lore raw

Re: [RFC PATCH v2 17/23] KVM: guest_memfd: Split for punch hole and private-to-shared conversion

On 8/7/2025 5:45 PM, Yan Zhao wrote:
[...]

Quoted 21 lines
virt/kvm/guest_memfd.c +7 -1
@@ -514,6 +554,8 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
  					   struct conversion_work *work,
  					   bool to_shared, pgoff_t *error_index)
  {
+	int ret = 0;
+
  	if (to_shared) {
  		struct list_head *gmem_list;
  		struct kvm_gmem *gmem;
@@ -522,19 +564,24 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
  		work_end = work->start + work->nr_pages;
 
  		gmem_list = &inode->i_mapping->i_private_list;
+		list_for_each_entry(gmem, gmem_list, entry) {
+			ret = kvm_gmem_split_private(gmem, work->start, work_end);
+			if (ret)
+				return ret;
+		}
  		list_for_each_entry(gmem, gmem_list, entry)
-			kvm_gmem_unmap_private(gmem, work->start, work_end);

virt/kvm/guest_memfd.c static int kvm_gmem_convert_should_proceed(struct inode *inode, · line 581

Quoted 10 lines
virt/kvm/guest_memfd.c +2 -1
+			kvm_gmem_zap(gmem, work->start, work_end, KVM_FILTER_PRIVATE);
  	} else {
  		unmap_mapping_pages(inode->i_mapping, work->start,
  				    work->nr_pages, false);
 
  		if (!kvm_gmem_has_safe_refcount(inode->i_mapping, work->start,
  						work->nr_pages, error_index)) {
-			return -EAGAIN;
+			ret = -EAGAIN;
  		}

Not from this patch.
When if statement breaks into two lines, are curly braces needed?

Quoted 6 lines
virt/kvm/guest_memfd.c +1 -1
  	}
 
-	return 0;
+	return ret;
  }
 

[...]

virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 1929

Quoted 7 lines
virt/kvm/guest_memfd.c +1 -2
@@ -1906,8 +1926,14 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
  	start = folio->index;
  	end = start + folio_nr_pages(folio);
 
-	list_for_each_entry(gmem, gmem_list, entry)
-		kvm_gmem_invalidate_begin_and_zap(gmem, start, end);
+	/* The size of the SEPT will not exceed the size of the folio */

To me, the comment alone without the context doesn't give a direct expression that split is not needed. If it's not too wordy, could you make it more informative?

Quoted 10 lines
virt/kvm/guest_memfd.c +7 -0
+	list_for_each_entry(gmem, gmem_list, entry) {
+		enum kvm_gfn_range_filter filter;
+
+		kvm_gmem_invalidate_begin(gmem, start, end);
+		filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED;
+		kvm_gmem_zap(gmem, start, end, filter);
+	}
 
  	/*
  	 * Do not truncate the range, what action is taken in response to the
author
#97 lore raw

Re: [RFC PATCH v2 17/23] KVM: guest_memfd: Split for punch hole and private-to-shared conversion

Quoted 27 lines · Binbin Wu

On 8/7/2025 5:45 PM, Yan Zhao wrote:
[...]

Quoted 23 lines
virt/kvm/guest_memfd.c +8 -1
@@ -514,6 +554,8 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
  					   struct conversion_work *work,
  					   bool to_shared, pgoff_t *error_index)
  {
+	int ret = 0;
+
  	if (to_shared) {
  		struct list_head *gmem_list;
  		struct kvm_gmem *gmem;
@@ -522,19 +564,24 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
  		work_end = work->start + work->nr_pages;
  		gmem_list = &inode->i_mapping->i_private_list;
+		list_for_each_entry(gmem, gmem_list, entry) {
+			ret = kvm_gmem_split_private(gmem, work->start, work_end);
+			if (ret)
+				return ret;
+		}
  		list_for_each_entry(gmem, gmem_list, entry)
-			kvm_gmem_unmap_private(gmem, work->start, work_end);
+			kvm_gmem_zap(gmem, work->start, work_end, KVM_FILTER_PRIVATE);
  	} else {
  		unmap_mapping_pages(inode->i_mapping, work->start,
  				    work->nr_pages, false);
Quoted 8 lines

virt/kvm/guest_memfd.c static int kvm_gmem_convert_should_proceed(struct inode *inode, · line 581

Quoted 5 lines
virt/kvm/guest_memfd.c +1 -1
  		if (!kvm_gmem_has_safe_refcount(inode->i_mapping, work->start,
  						work->nr_pages, error_index)) {
-			return -EAGAIN;
+			ret = -EAGAIN;
  		}

Not from this patch.
When if statement breaks into two lines, are curly braces needed?

Hmm, either one (with or without curly braces) can pass the check of "scripts/checkpatch.pl --strict".

Quoted 6 lines
Quoted 5 lines
virt/kvm/guest_memfd.c +1 -1
  	}
-	return 0;
+	return ret;
  }

[...]

Quoted 8 lines

virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 1928

Quoted 6 lines
virt/kvm/guest_memfd.c +1 -2
@@ -1906,8 +1926,14 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
  	start = folio->index;
  	end = start + folio_nr_pages(folio);
-	list_for_each_entry(gmem, gmem_list, entry)
-		kvm_gmem_invalidate_begin_and_zap(gmem, start, end);
+	/* The size of the SEPT will not exceed the size of the folio */

To me, the comment alone without the context doesn't give a direct expression that split is not needed. If it's not too wordy, could you make it more informative?

What about:
The zap is limited to the range covered by a single folio.
As a S-EPT leaf entry can't cover a range larger than its backend folio size, the zap can't cross two S-EPT leaf entries. So, no split is required.

Quoted 11 lines
Quoted 10 lines
virt/kvm/guest_memfd.c +7 -0
+	list_for_each_entry(gmem, gmem_list, entry) {
+		enum kvm_gfn_range_filter filter;
+
+		kvm_gmem_invalidate_begin(gmem, start, end);
+		filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED;
+		kvm_gmem_zap(gmem, start, end, filter);
+	}
  	/*
  	 * Do not truncate the range, what action is taken in response to the
author
#98 lore raw

Re: [RFC PATCH v2 17/23] KVM: guest_memfd: Split for punch hole and private-to-shared conversion

Quoted 9 lines · Yan Zhao
Quoted 8 lines · Binbin Wu

virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 1928

Quoted 6 lines
virt/kvm/guest_memfd.c +1 -2
@@ -1906,8 +1926,14 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
  	start = folio->index;
  	end = start + folio_nr_pages(folio);
-	list_for_each_entry(gmem, gmem_list, entry)
-		kvm_gmem_invalidate_begin_and_zap(gmem, start, end);
+	/* The size of the SEPT will not exceed the size of the folio */

To me, the comment alone without the context doesn't give a direct expression that split is not needed. If it's not too wordy, could you make it more informative?

Quoted 4 lines

What about:
The zap is limited to the range covered by a single folio.
As a S-EPT leaf entry can't cover a range larger than its backend folio size, the zap can't cross two S-EPT leaf entries. So, no split is required.

Sorry, my brain just froze.
Should just be:

As a leaf SPTE can't cover a range larger than its backend folio size, no splitting is required before the zap.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
virt/kvm/guest_memfd.c +7 -0
+	list_for_each_entry(gmem, gmem_list, entry) {
+		enum kvm_gfn_range_filter filter;
+
+		kvm_gmem_invalidate_begin(gmem, start, end);
+		filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED;
+		kvm_gmem_zap(gmem, start, end, filter);
+	}
  	/*
  	 * Do not truncate the range, what action is taken in response to the
#99 lore raw

Re: [RFC PATCH v2 17/23] KVM: guest_memfd: Split for punch hole and private-to-shared conversion

In reply to Yan Zhao

Yan Zhao <[email protected]> writes:

Thanks Yan! Just got around to looking at this, sorry about the delay!

Quoted 10 lines

In TDX, private page tables require precise zapping because faulting back the zapped mappings necessitates the guest's re-acceptance. Therefore, before performing a zap for hole punching and private-to-shared conversions, huge leafs that cross the boundary of the zapping GFN range in the mirror page table must be split.

Splitting may result in an error. If this happens, hole punching and private-to-shared conversion should bail out early and return an error to userspace.

Quoted 9 lines

Splitting is not necessary for kvm_gmem_release() since the entire page table is being zapped, nor for kvm_gmem_error_folio() as an SPTE must not map more than one physical folio.

Therefore, in this patch,
- break kvm_gmem_invalidate_begin_and_zap() into kvm_gmem_invalidate_begin() and kvm_gmem_zap() and have kvm_gmem_release() and kvm_gmem_error_folio() to invoke them.

I think perhaps separating invalidate and zip could be a separate patch from adding the split step into the flow, that would make this patch smaller and easier to review.

No action required from you for now, I have the the above part in a separate patch already (not yet posted).

Quoted 4 lines

- have kvm_gmem_punch_hole() to invoke kvm_gmem_invalidate_begin(), kvm_gmem_split_private(), and kvm_gmem_zap().
Bail out if kvm_gmem_split_private() returns error.

IIUC the current upstream position is that hole punching will not be permitted for ranges smaller than the page size for the entire guest_memfd.

Hence no splitting required during hole punch?

+ 4K guest_memfd: no splitting required since the EPT entries will not be larger than 4K anyway
+ 2M and 1G (x86) guest_memfd: no splitting required since the entire EPT entry will have to go away for valid ranges (valid ranges are either 2M or 1G anyway)

Does that sound right?

Quoted 20 lines

- drop the old kvm_gmem_unmap_private() and have private-to-shared conversion to invoke kvm_gmem_split_private() and kvm_gmem_zap() instead. Bail out if kvm_gmem_split_private() returns error.

[...snip...]

virt/kvm/guest_memfd.c +2 -0
@@ -514,6 +554,8 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
 					   struct conversion_work *work,
 					   bool to_shared, pgoff_t *error_index)
 {
+	int ret = 0;
+
 	if (to_shared) {
 		struct list_head *gmem_list;
 		struct kvm_gmem *gmem;

virt/kvm/guest_memfd.c static int kvm_gmem_convert_should_proceed(struct inode *inode, · line 571

Quoted 8 lines
virt/kvm/guest_memfd.c +5 -0
@@ -522,19 +564,24 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
 		work_end = work->start + work->nr_pages;
 
 		gmem_list = &inode->i_mapping->i_private_list;
+		list_for_each_entry(gmem, gmem_list, entry) {
+			ret = kvm_gmem_split_private(gmem, work->start, work_end);
+			if (ret)
+				return ret;
+		}

Will be refactoring the conversion steps a little for the next version of this series, hence I'd like to ask about the requirements before doing splitting.

The requirement is to split before zapping, right? Other than that we technically don't need to split before checking for a safe refcount, right?

Quoted 22 lines
virt/kvm/guest_memfd.c +3 -3
 		list_for_each_entry(gmem, gmem_list, entry)
-			kvm_gmem_unmap_private(gmem, work->start, work_end);
+			kvm_gmem_zap(gmem, work->start, work_end, KVM_FILTER_PRIVATE);
 	} else {
 		unmap_mapping_pages(inode->i_mapping, work->start,
 				    work->nr_pages, false);
 
 		if (!kvm_gmem_has_safe_refcount(inode->i_mapping, work->start,
 						work->nr_pages, error_index)) {
-			return -EAGAIN;
+			ret = -EAGAIN;
 		}
 	}
 
-	return 0;
+	return ret;
 }
 
 
[...snip...]

virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 1929

Quoted 6 lines
virt/kvm/guest_memfd.c +1 -2
@@ -1906,8 +1926,14 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	start = folio->index;
 	end = start + folio_nr_pages(folio);
 
-	list_for_each_entry(gmem, gmem_list, entry)
-		kvm_gmem_invalidate_begin_and_zap(gmem, start, end);
+	/* The size of the SEPT will not exceed the size of the folio */

I think splitting might be required here, but that depends on whether we want to unmap just a part of the huge folio or whether we want to unmap the entire folio.

Lots of open questions on memory failure handling, but for now I think this makes sense.

Quoted 12 lines
virt/kvm/guest_memfd.c +7 -0
+	list_for_each_entry(gmem, gmem_list, entry) {
+		enum kvm_gfn_range_filter filter;
+
+		kvm_gmem_invalidate_begin(gmem, start, end);
+		filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED;
+		kvm_gmem_zap(gmem, start, end, filter);
+	}
 
 	/*
 	 * Do not truncate the range, what action is taken in response to the
Signature · 1 line
2.43.2
author
#100 lore raw

Re: [RFC PATCH v2 17/23] KVM: guest_memfd: Split for punch hole and private-to-shared conversion

Sorry for the delay. Just back from the vacation.

Quoted 31 lines · Ackerley Tng

Yan Zhao <[email protected]> writes:

Thanks Yan! Just got around to looking at this, sorry about the delay!

Quoted 10 lines

In TDX, private page tables require precise zapping because faulting back the zapped mappings necessitates the guest's re-acceptance. Therefore, before performing a zap for hole punching and private-to-shared conversions, huge leafs that cross the boundary of the zapping GFN range in the mirror page table must be split.

Splitting may result in an error. If this happens, hole punching and private-to-shared conversion should bail out early and return an error to userspace.

Quoted 9 lines

Splitting is not necessary for kvm_gmem_release() since the entire page table is being zapped, nor for kvm_gmem_error_folio() as an SPTE must not map more than one physical folio.

Therefore, in this patch,
- break kvm_gmem_invalidate_begin_and_zap() into kvm_gmem_invalidate_begin() and kvm_gmem_zap() and have kvm_gmem_release() and kvm_gmem_error_folio() to invoke them.

I think perhaps separating invalidate and zip could be a separate patch from adding the split step into the flow, that would make this patch smaller and easier to review.

No action required from you for now, I have the the above part in a separate patch already (not yet posted).

Quoted 8 lines
Quoted 4 lines

- have kvm_gmem_punch_hole() to invoke kvm_gmem_invalidate_begin(), kvm_gmem_split_private(), and kvm_gmem_zap().
Bail out if kvm_gmem_split_private() returns error.

IIUC the current upstream position is that hole punching will not be permitted for ranges smaller than the page size for the entire guest_memfd.

In hugetlbfs_fallocate(), the punch hole ranges are hpage size aligned.

    start = offset >> hpage_shift;
    end = (offset + len + hpage_size - 1) >> hpage_shift;

However, in the guest_memfd (at least in v2), the punch hole ranges are just page aligned.

    pgoff_t start = offset >> PAGE_SHIFT;
    pgoff_t end = (offset + len) >> PAGE_SHIFT;
(Note, I noticed that the range calculation for invalidation is not the same as
that in kvm_gmem_truncate_inode_range(), where: 
    full_hpage_start = round_up(start, nr_per_huge_page);
    full_hpage_end = round_down(end, nr_per_huge_page);

We should probably align these two implementations for consistency).

Quoted 9 lines

Hence no splitting required during hole punch?

+ 4K guest_memfd: no splitting required since the EPT entries will not be larger than 4K anyway
+ 2M and 1G (x86) guest_memfd: no splitting required since the entire EPT entry will have to go away for valid ranges (valid ranges are either 2M or 1G anyway)

Does that sound right?

If future guest_memfd code could align the punch hole ranges to page size for the entire guest_memfd, I think it's ok.

Quoted 33 lines
Quoted 20 lines

- drop the old kvm_gmem_unmap_private() and have private-to-shared conversion to invoke kvm_gmem_split_private() and kvm_gmem_zap() instead. Bail out if kvm_gmem_split_private() returns error.

[...snip...]

virt/kvm/guest_memfd.c +2 -0
@@ -514,6 +554,8 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
 					   struct conversion_work *work,
 					   bool to_shared, pgoff_t *error_index)
 {
+	int ret = 0;
+
 	if (to_shared) {
 		struct list_head *gmem_list;
 		struct kvm_gmem *gmem;

virt/kvm/guest_memfd.c static int kvm_gmem_convert_should_proceed(struct inode *inode, · line 571

Quoted 8 lines
virt/kvm/guest_memfd.c +5 -0
@@ -522,19 +564,24 @@ static int kvm_gmem_convert_should_proceed(struct inode *inode,
 		work_end = work->start + work->nr_pages;
 
 		gmem_list = &inode->i_mapping->i_private_list;
+		list_for_each_entry(gmem, gmem_list, entry) {
+			ret = kvm_gmem_split_private(gmem, work->start, work_end);
+			if (ret)
+				return ret;
+		}

Will be refactoring the conversion steps a little for the next version of this series, hence I'd like to ask about the requirements before doing splitting.

Quoted 2 lines

The requirement is to split before zapping, right? Other than that we technically don't need to split before checking for a safe refcount, right?

Yes, the split is for private-to-shared conversion. TDX will not hold page refcount for private pages any more.

Quoted 25 lines
Quoted 25 lines
virt/kvm/guest_memfd.c +3 -3
 		list_for_each_entry(gmem, gmem_list, entry)
-			kvm_gmem_unmap_private(gmem, work->start, work_end);
+			kvm_gmem_zap(gmem, work->start, work_end, KVM_FILTER_PRIVATE);
 	} else {
 		unmap_mapping_pages(inode->i_mapping, work->start,
 				    work->nr_pages, false);
 
 		if (!kvm_gmem_has_safe_refcount(inode->i_mapping, work->start,
 						work->nr_pages, error_index)) {
-			return -EAGAIN;
+			ret = -EAGAIN;
 		}
 	}
 
-	return 0;
+	return ret;
 }
 
 
[...snip...]
@@ -1906,8 +1926,14 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	start = folio->index;
 	end = start + folio_nr_pages(folio);
 
Quoted 7 lines

virt/kvm/guest_memfd.c static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol · line 1929

Quoted 3 lines
virt/kvm/guest_memfd.c +1 -2
-	list_for_each_entry(gmem, gmem_list, entry)
-		kvm_gmem_invalidate_begin_and_zap(gmem, start, end);
+	/* The size of the SEPT will not exceed the size of the folio */

I think splitting might be required here, but that depends on whether we want to unmap just a part of the huge folio or whether we want to unmap the entire folio.

Ok. When that occurs, we can do the split according to the partial unmap range info.

Quoted 15 lines

Lots of open questions on memory failure handling, but for now I think this makes sense.

Quoted 12 lines
virt/kvm/guest_memfd.c +7 -0
+	list_for_each_entry(gmem, gmem_list, entry) {
+		enum kvm_gfn_range_filter filter;
+
+		kvm_gmem_invalidate_begin(gmem, start, end);
+		filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED;
+		kvm_gmem_zap(gmem, start, end, filter);
+	}
 
 	/*
 	 * Do not truncate the range, what action is taken in response to the
Signature · 1 line
2.43.2
#101 lore raw

Re: [RFC PATCH v2 17/23] KVM: guest_memfd: Split for punch hole and private-to-shared conversion

In reply to Yan Zhao

Yan Zhao <[email protected]> writes:

I was looking deeper into this patch since on my WIP tree I already had the invalidate and zap steps separated out and had to do more to rebase this patch :)

Quoted 2 lines

In TDX, private page tables require precise zapping because faulting back the zapped mappings necessitates the guest's re-acceptance.

I feel that this statement could be better phrased because all zapped mappings require re-acceptance, not just anything related to precise zapping. Would this be better:

On private-to-shared conversions, page table entries must be zapped from the Secure EPTs. Any pages mapped into Secure EPTs must be accepted by the guest before they are used.

Hence, care must be taken to only precisely zap ranges requested for private-to-shared conversion, since the guest is only prepared to re-accept precisely the ranges it requested for conversion.

The guest may request to convert ranges not aligned with private page table entry boundaries. To precisely zap these ranges, huge leaves that span the boundaries of the requested ranges must be split into smaller leaves, so that the split, smaller leaves now align with the requested range for zapping.

Quoted 5 lines

Therefore,
before performing a zap for hole punching and private-to-shared conversions, huge leafs that cross the boundary of the zapping GFN range in the mirror page table must be split.

Quoted 8 lines

Splitting may result in an error. If this happens, hole punching and private-to-shared conversion should bail out early and return an error to userspace.

Splitting is not necessary for kvm_gmem_release() since the entire page table is being zapped, nor for kvm_gmem_error_folio() as an SPTE must not map more than one physical folio.

I think splitting is not necessary as long as aligned page table entries are zapped. Splitting is also not necessary if the entire page table is zapped but that's a superset of zapping aligned page table entries. (Probably just a typo on your side.) Here's my attempt at rephrasing this:

Splitting is not necessary for the cases where only aligned page table entries are zapped, such as during kvm_gmem_release() where the entire guest_memfd worth of memory is zapped, nor for truncation, where truncation of pages within a huge folio is not allowed.

Quoted 30 lines

Therefore, in this patch,
- break kvm_gmem_invalidate_begin_and_zap() into kvm_gmem_invalidate_begin() and kvm_gmem_zap() and have kvm_gmem_release() and kvm_gmem_error_folio() to invoke them.

- have kvm_gmem_punch_hole() to invoke kvm_gmem_invalidate_begin(), kvm_gmem_split_private(), and kvm_gmem_zap().
Bail out if kvm_gmem_split_private() returns error.

- drop the old kvm_gmem_unmap_private() and have private-to-shared conversion to invoke kvm_gmem_split_private() and kvm_gmem_zap() instead. Bail out if kvm_gmem_split_private() returns error.

---
RFC v2:
- Rebased to [1]. As changes in this patch are gmem specific, they may need to be updated if the implementation in [1] changes.
- Update kvm_split_boundary_leafs() to kvm_split_cross_boundary_leafs() and invoke it before kvm_gmem_punch_hole() and private-to-shared conversion.

[1] https://lore.kernel.org/all/[email protected]/

RFC v1:
- new patch.

[...snip...]

author
#102 lore raw

Re: [RFC PATCH v2 17/23] KVM: guest_memfd: Split for punch hole and private-to-shared conversion

Quoted 21 lines · Ackerley Tng

Yan Zhao <[email protected]> writes:

I was looking deeper into this patch since on my WIP tree I already had the invalidate and zap steps separated out and had to do more to rebase this patch :)

Quoted 2 lines

In TDX, private page tables require precise zapping because faulting back the zapped mappings necessitates the guest's re-acceptance.

I feel that this statement could be better phrased because all zapped mappings require re-acceptance, not just anything related to precise zapping. Would this be better:

On private-to-shared conversions, page table entries must be zapped from the Secure EPTs. Any pages mapped into Secure EPTs must be accepted by the guest before they are used.

Hence, care must be taken to only precisely zap ranges requested for private-to-shared conversion, since the guest is only prepared to re-accept precisely the ranges it requested for conversion.

Quoted 5 lines

The guest may request to convert ranges not aligned with private page table entry boundaries. To precisely zap these ranges, huge leaves that span the boundaries of the requested ranges must be split into smaller leaves, so that the split, smaller leaves now align with the requested range for zapping.

LGTM. Thanks!

Quoted 14 lines
Quoted 5 lines

Therefore,
before performing a zap for hole punching and private-to-shared conversions, huge leafs that cross the boundary of the zapping GFN range in the mirror page table must be split.

Quoted 8 lines

Splitting may result in an error. If this happens, hole punching and private-to-shared conversion should bail out early and return an error to userspace.

Splitting is not necessary for kvm_gmem_release() since the entire page table is being zapped, nor for kvm_gmem_error_folio() as an SPTE must not map more than one physical folio.

Quoted 4 lines

I think splitting is not necessary as long as aligned page table entries are zapped. Splitting is also not necessary if the entire page table is zapped but that's a superset of zapping aligned page table entries. (Probably just a typo on your side.) Here's my attempt at

what is the typo you are referring to?

Quoted 4 lines

rephrasing this:

Splitting is not necessary for the cases where only aligned page table entries are zapped, such as during kvm_gmem_release() where

By "page table entries", you mean SPTEs, i.e., entries in the secondary MMU, right?

Quoted 3 lines

the entire guest_memfd worth of memory is zapped, nor for truncation, where truncation of pages within a huge folio is not allowed.

I think that "splitting is not required for truncation" is valid only based on KVM's implementation where "an SPTE must not map more than one physical folio". i.e., the SPTE entry size is <= folio size.

If KVM were implemented differently where one SPTE could cover multiple folios (similar to IOMMU SLTP entries for shared memory, though this is unlikely to happen), splitting would still be required before truncation.

author
#103 lore raw

RFC PATCH v2 18/23 x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

In reply to Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
 arch/x86/include/asm/tdx.h  |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 19 +++++++++++++------
 2 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/include/asm/tdx.h +1 -0
index f1bd74348b34..c058a82d4a97 100644
@@ -15,6 +15,7 @@
 
 /* Bit definitions of TDX_FEATURES0 metadata field */
 #define TDX_FEATURES0_NO_RBP_MOD		BIT_ULL(18)
+#define TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC	BIT_ULL(23)
 #define TDX_FEATURES0_DYNAMIC_PAMT		BIT_ULL(36)
 
 #ifndef __ASSEMBLER__
arch/x86/virt/vmx/tdx/tdx.c +13 -6
index 9ed585bde062..b7a0ee0f4a50 100644
@@ -1648,14 +1648,13 @@ static inline u64 tdx_tdvpr_pa(struct tdx_vp *td)
 	return page_to_phys(td->tdvpr_page);
 }
 
-/*
- * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether
- * a CLFLUSH of pages is required before handing them to the TDX module.
- * Be conservative and make the code simpler by doing the CLFLUSH
- * unconditionally.
- */
 static void tdx_clflush_page(struct page *page)
 {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
+
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return;
+
 	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
 }
 
@@ -2030,8 +2029,12 @@ EXPORT_SYMBOL_GPL(tdh_phymem_cache_wb);
 
 u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td)
 {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
 	struct tdx_module_args args = {};
 
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+
 	args.rcx = mk_keyed_paddr(tdx_global_keyid, td->tdr_page);
 
 	return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
@@ -2041,10 +2044,14 @@ EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_tdr);
 u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
 				unsigned long start_idx, unsigned long npages)
 {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
 	struct page *start = folio_page(folio, start_idx);
 	struct tdx_module_args args = {};
 	u64 err;
 
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+
 	if (start_idx + npages > folio_nr_pages(folio))
 		return TDX_OPERAND_INVALID;
 
Signature · 1 line
2.43.2
#104 lore raw

Re: [RFC PATCH v2 18/23] x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

Quoted 50 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
 arch/x86/include/asm/tdx.h  |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 19 +++++++++++++------
 2 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/include/asm/tdx.h +1 -0
index f1bd74348b34..c058a82d4a97 100644
@@ -15,6 +15,7 @@
 
 /* Bit definitions of TDX_FEATURES0 metadata field */
 #define TDX_FEATURES0_NO_RBP_MOD               BIT_ULL(18)
+#define TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC     BIT_ULL(23)
 #define TDX_FEATURES0_DYNAMIC_PAMT             BIT_ULL(36)
 
 #ifndef __ASSEMBLER__
arch/x86/virt/vmx/tdx/tdx.c +0 -2
index 9ed585bde062..b7a0ee0f4a50 100644
@@ -1648,14 +1648,13 @@ static inline u64 tdx_tdvpr_pa(struct tdx_vp *td)
        return page_to_phys(td->tdvpr_page);
 }
 
-/*
- * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdvpr_pa(struct tdx_vp *td) · line 1656

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -4
- * a CLFLUSH of pages is required before handing them to the TDX module.
- * Be conservative and make the code simpler by doing the CLFLUSH
- * unconditionally.
- */
 static void tdx_clflush_page(struct page *page)
 {
+       u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
+
+       if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+               return;

Isn't the logic here and below reversed? If TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC bit is set, we want to perform the clflush()

Quoted 36 lines
arch/x86/virt/vmx/tdx/tdx.c +9 -1
+
        clflush_cache_range(page_to_virt(page), PAGE_SIZE);
 }
 
@@ -2030,8 +2029,12 @@ EXPORT_SYMBOL_GPL(tdh_phymem_cache_wb);
 
 u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td)
 {
+       u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
        struct tdx_module_args args = {};
 
+       if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+               return 0;
+
        args.rcx = mk_keyed_paddr(tdx_global_keyid, td->tdr_page);
 
        return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
@@ -2041,10 +2044,14 @@ EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_tdr);
 u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
                                unsigned long start_idx, unsigned long npages)
 {
+       u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
        struct page *start = folio_page(folio, start_idx);
        struct tdx_module_args args = {};
        u64 err;
 
+       if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+               return 0;
+
        if (start_idx + npages > folio_nr_pages(folio))
                return TDX_OPERAND_INVALID;
 
--

2.43.2

author
#105 lore raw

Re: [RFC PATCH v2 18/23] x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

Quoted 8 lines · Sagi Shahar
Quoted 7 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Quoted 5 lines
Quoted 5 lines

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

I actually couldn't observe this failure in my side with DPAMT + hugepage (without shutdown optimization).

Quoted 35 lines
Quoted 35 lines

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
 arch/x86/include/asm/tdx.h  |  1 +
 arch/x86/virt/vmx/tdx/tdx.c | 19 +++++++++++++------
 2 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/include/asm/tdx.h +1 -0
index f1bd74348b34..c058a82d4a97 100644
@@ -15,6 +15,7 @@
 
 /* Bit definitions of TDX_FEATURES0 metadata field */
 #define TDX_FEATURES0_NO_RBP_MOD               BIT_ULL(18)
+#define TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC     BIT_ULL(23)
 #define TDX_FEATURES0_DYNAMIC_PAMT             BIT_ULL(36)
 
 #ifndef __ASSEMBLER__
arch/x86/virt/vmx/tdx/tdx.c +0 -0
index 9ed585bde062..b7a0ee0f4a50 100644
@@ -1648,14 +1648,13 @@ static inline u64 tdx_tdvpr_pa(struct tdx_vp *td)
        return page_to_phys(td->tdvpr_page);
 }
 
Quoted 16 lines

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdvpr_pa(struct tdx_vp *td) · line 1656

Quoted 12 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -6
-/*
- * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether
- * a CLFLUSH of pages is required before handing them to the TDX module.
- * Be conservative and make the code simpler by doing the CLFLUSH
- * unconditionally.
- */
 static void tdx_clflush_page(struct page *page)
 {
+       u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
+
+       if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+               return;

Isn't the logic here and below reversed? If TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC bit is set, we want to perform the clflush()

Yes, I think so.

As my test machine has boot_cpu_has_bug(X86_BUG_TDX_PW_MCE) returning true, I thought it was right to perform clflush() and overlooked this logical error.

Quoted 36 lines
Quoted 35 lines
arch/x86/virt/vmx/tdx/tdx.c +8 -1
        clflush_cache_range(page_to_virt(page), PAGE_SIZE);
 }
 
@@ -2030,8 +2029,12 @@ EXPORT_SYMBOL_GPL(tdh_phymem_cache_wb);
 
 u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td)
 {
+       u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
        struct tdx_module_args args = {};
 
+       if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+               return 0;
+
        args.rcx = mk_keyed_paddr(tdx_global_keyid, td->tdr_page);
 
        return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
@@ -2041,10 +2044,14 @@ EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_tdr);
 u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
                                unsigned long start_idx, unsigned long npages)
 {
+       u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
        struct page *start = folio_page(folio, start_idx);
        struct tdx_module_args args = {};
        u64 err;
 
+       if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+               return 0;
+
        if (start_idx + npages > folio_nr_pages(folio))
                return TDX_OPERAND_INVALID;
 
--

2.43.2

#106 lore raw

Re: [RFC PATCH v2 18/23] x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

In reply to Yan Zhao
Quoted 48 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
  arch/x86/include/asm/tdx.h  |  1 +
  arch/x86/virt/vmx/tdx/tdx.c | 19 +++++++++++++------
  2 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/include/asm/tdx.h +1 -0
index f1bd74348b34..c058a82d4a97 100644
@@ -15,6 +15,7 @@
 
  /* Bit definitions of TDX_FEATURES0 metadata field */
  #define TDX_FEATURES0_NO_RBP_MOD		BIT_ULL(18)
+#define TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC	BIT_ULL(23)
  #define TDX_FEATURES0_DYNAMIC_PAMT		BIT_ULL(36)
 
  #ifndef __ASSEMBLER__
arch/x86/virt/vmx/tdx/tdx.c +0 -1
index 9ed585bde062..b7a0ee0f4a50 100644
@@ -1648,14 +1648,13 @@ static inline u64 tdx_tdvpr_pa(struct tdx_vp *td)
  	return page_to_phys(td->tdvpr_page);
  }
 
-/*

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdvpr_pa(struct tdx_vp *td) · line 1655

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -5
- * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether
- * a CLFLUSH of pages is required before handing them to the TDX module.
- * Be conservative and make the code simpler by doing the CLFLUSH
- * unconditionally.
- */
  static void tdx_clflush_page(struct page *page)
  {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
+
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)

According to the cover letter, if TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC is enabled, an explicit cache flush is necessary.
Shouldn't this and below be:
if (!(tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC))

Quoted 33 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+		return;
+
  	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
  }
 
@@ -2030,8 +2029,12 @@ EXPORT_SYMBOL_GPL(tdh_phymem_cache_wb);
 
  u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td)
  {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
  	struct tdx_module_args args = {};
 
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+
  	args.rcx = mk_keyed_paddr(tdx_global_keyid, td->tdr_page);
 
  	return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
@@ -2041,10 +2044,14 @@ EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_tdr);
  u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
  				unsigned long start_idx, unsigned long npages)
  {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
  	struct page *start = folio_page(folio, start_idx);
  	struct tdx_module_args args = {};
  	u64 err;
 
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+
  	if (start_idx + npages > folio_nr_pages(folio))
  		return TDX_OPERAND_INVALID;
 
author
#107 lore raw

Re: [RFC PATCH v2 18/23] x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

Quoted 55 lines · Binbin Wu
Quoted 52 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
  arch/x86/include/asm/tdx.h  |  1 +
  arch/x86/virt/vmx/tdx/tdx.c | 19 +++++++++++++------
  2 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/include/asm/tdx.h +1 -0
index f1bd74348b34..c058a82d4a97 100644
@@ -15,6 +15,7 @@
  /* Bit definitions of TDX_FEATURES0 metadata field */
  #define TDX_FEATURES0_NO_RBP_MOD		BIT_ULL(18)
+#define TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC	BIT_ULL(23)
  #define TDX_FEATURES0_DYNAMIC_PAMT		BIT_ULL(36)
  #ifndef __ASSEMBLER__
arch/x86/virt/vmx/tdx/tdx.c +0 -6
index 9ed585bde062..b7a0ee0f4a50 100644
@@ -1648,14 +1648,13 @@ static inline u64 tdx_tdvpr_pa(struct tdx_vp *td)
  	return page_to_phys(td->tdvpr_page);
  }
-/*
- * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether
- * a CLFLUSH of pages is required before handing them to the TDX module.
- * Be conservative and make the code simpler by doing the CLFLUSH
- * unconditionally.
- */
  static void tdx_clflush_page(struct page *page)
  {
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdvpr_pa(struct tdx_vp *td) · line 1655

Quoted 3 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
+
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)

According to the cover letter, if TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC is enabled, an explicit cache flush is necessary.
Shouldn't this and below be:
if (!(tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC))

Right, Sagi also reported it.
https://lore.kernel.org/kvm/CAAhR5DEZZfX0=9QwBrXhC+1fp1Z0w4Xbb3mXcn0OuW+45tsLwA@mail.gmail.com/

Quoted 28 lines
Quoted 27 lines
arch/x86/virt/vmx/tdx/tdx.c +10 -0
+		return;
+
  	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
  }
@@ -2030,8 +2029,12 @@ EXPORT_SYMBOL_GPL(tdh_phymem_cache_wb);
  u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td)
  {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
  	struct tdx_module_args args = {};
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+
  	args.rcx = mk_keyed_paddr(tdx_global_keyid, td->tdr_page);
  	return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
@@ -2041,10 +2044,14 @@ EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_tdr);
  u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
  				unsigned long start_idx, unsigned long npages)
  {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
  	struct page *start = folio_page(folio, start_idx);
  	struct tdx_module_args args = {};
  	u64 err;
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+
  	if (start_idx + npages > folio_nr_pages(folio))
  		return TDX_OPERAND_INVALID;
#108 lore raw

Re: [RFC PATCH v2 18/23] x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

Quoted 65 lines · Yan Zhao
Quoted 53 lines · Binbin Wu
Quoted 51 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
   arch/x86/include/asm/tdx.h  |  1 +
   arch/x86/virt/vmx/tdx/tdx.c | 19 +++++++++++++------
   2 files changed, 14 insertions(+), 6 deletions(-)
arch/x86/include/asm/tdx.h +1 -0
index f1bd74348b34..c058a82d4a97 100644
@@ -15,6 +15,7 @@
   /* Bit definitions of TDX_FEATURES0 metadata field */
   #define TDX_FEATURES0_NO_RBP_MOD		BIT_ULL(18)
+#define TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC	BIT_ULL(23)
   #define TDX_FEATURES0_DYNAMIC_PAMT		BIT_ULL(36)
   #ifndef __ASSEMBLER__
arch/x86/virt/vmx/tdx/tdx.c +0 -6
index 9ed585bde062..b7a0ee0f4a50 100644
@@ -1648,14 +1648,13 @@ static inline u64 tdx_tdvpr_pa(struct tdx_vp *td)
   	return page_to_phys(td->tdvpr_page);
   }
-/*
- * The TDX module exposes a CLFLUSH_BEFORE_ALLOC bit to specify whether
- * a CLFLUSH of pages is required before handing them to the TDX module.
- * Be conservative and make the code simpler by doing the CLFLUSH
- * unconditionally.
- */
   static void tdx_clflush_page(struct page *page)
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c static inline u64 tdx_tdvpr_pa(struct tdx_vp *td) · line 1655

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +3 -0
   {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
+
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)

According to the cover letter, if TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC is enabled, an explicit cache flush is necessary.
Shouldn't this and below be:
if (!(tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC))

Right, Sagi also reported it.
https://lore.kernel.org/kvm/CAAhR5DEZZfX0=9QwBrXhC+1fp1Z0w4Xbb3mXcn0OuW+45tsLwA@mail.gmail.com/

Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +2 -0
+		return;
+
   	clflush_cache_range(page_to_virt(page), PAGE_SIZE);
   }
Quoted 8 lines

arch/x86/virt/vmx/tdx/tdx.c EXPORT_SYMBOL_GPL(tdh_phymem_cache_wb); · line 2035

Quoted 8 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -0
@@ -2030,8 +2029,12 @@ EXPORT_SYMBOL_GPL(tdh_phymem_cache_wb);
   u64 tdh_phymem_page_wbinvd_tdr(struct tdx_td *td)
   {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
   	struct tdx_module_args args = {};
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+

According to the description of TDX module base spec (348549006), CLFLUSH_BEFORE_ALLOC is related to clfush requirement before adding a page to TDX module.

If it also applies to the pages returned back from TDX module, I think it needs to be called out somewhere.

Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
arch/x86/virt/vmx/tdx/tdx.c +4 -0
   	args.rcx = mk_keyed_paddr(tdx_global_keyid, td->tdr_page);
   	return seamcall(TDH_PHYMEM_PAGE_WBINVD, &args);
@@ -2041,10 +2044,14 @@ EXPORT_SYMBOL_GPL(tdh_phymem_page_wbinvd_tdr);
   u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, struct folio *folio,
   				unsigned long start_idx, unsigned long npages)
   {
+	u64 tdx_features0 = tdx_sysinfo.features.tdx_features0;
   	struct page *start = folio_page(folio, start_idx);
   	struct tdx_module_args args = {};
   	u64 err;
+	if (tdx_features0 & TDX_FEATURES0_CLFLUSH_BEFORE_ALLOC)
+		return 0;
+
   	if (start_idx + npages > folio_nr_pages(folio))
   		return TDX_OPERAND_INVALID;
#109 lore raw

Re: [RFC PATCH v2 18/23] x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

In reply to Yan Zhao
Quoted 12 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

Quoted 5 lines

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

I think I mentioned this on some version of this patch already, but during the base series we decided to assume CLFLUSH_BEFORE_ALLOC was always set for simplicity. Let's try to be consistent.

Why prepare for some future TDX module that sets CLFLUSH_BEFORE_ALLOC *and* adds new support for at larger page sizes TDH.PHYMEM.PAGE.WBINVD? It almost seems like this is working around a bug.

author
#110 lore raw

Re: [RFC PATCH v2 18/23] x86/virt/tdx: Do not perform cache flushes unless CLFLUSH_BEFORE_ALLOC is set

Quoted 7 lines · Rick P
Quoted 6 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

The TDX module enumerates with a TDX_FEATURES0 bit if an explicit cache flush is necessary when switching KeyID for a page, like before handing the page over to a TD.

Quoted 5 lines
Quoted 5 lines

Currently, none of the TDX-capable platforms have this bit enabled.

Moreover, cache flushing with TDH.PHYMEM.PAGE.WBINVD fails if Dynamic PAMT is active and the target page is not 4k. The SEAMCALL only supports 4k pages and will fail if there is no PAMT_4K for the HPA.

Back to when Kirill was enabling DPAMT for huge pages, TDH.PHYMEM.PAGE.WBINVD did fail in this scenario.
However, this has been fixed in TDX_1.5.20, which is why I didn't encounter this issue when posting this series.

Quoted 9 lines
Quoted 5 lines

Avoid performing these cache flushes unless the CLFLUSH_BEFORE_ALLOC bit of TDX_FEATURES0 is set.

I think I mentioned this on some version of this patch already, but during the base series we decided to assume CLFLUSH_BEFORE_ALLOC was always set for simplicity. Let's try to be consistent.

Right, though CLFLUSH_BEFORE_ALLOC is always false in all current TDX modules, linux conservatively assumes it's always true.

Quoted 3 lines

Why prepare for some future TDX module that sets CLFLUSH_BEFORE_ALLOC *and* adds new support for at larger page sizes TDH.PHYMEM.PAGE.WBINVD? It almost seems like this is working around a bug.

As the TDX module bug is gone, let's drop this patch to be consistent with the policy of assuming CLFLUSH_BEFORE_ALLOC is always true.

author
#111 lore raw

RFC PATCH v2 19/23 KVM: TDX: Pass down pfn to split_external_spt()

In reply to Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Pass down pfn to kvm_x86_ops::split_external_spt(). It is required for handling Dynamic PAMT in tdx_sept_split_private_spt().

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
 arch/x86/include/asm/kvm_host.h | 3 ++-
 arch/x86/kvm/mmu/tdp_mmu.c      | 6 +++++-
 arch/x86/kvm/vmx/tdx.c          | 3 ++-
 3 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -1
index 6cb5b422dd1d..6b6c46c27390 100644
@@ -1841,7 +1841,8 @@ struct kvm_x86_ops {
 
 	/* Split the external page table into smaller page tables */
 	int (*split_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
-				  void *external_spt, bool mmu_lock_shared);
+				  kvm_pfn_t pfn_for_gfn, void *external_spt,
+				  bool mmu_lock_shared);
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/tdp_mmu.c +5 -1
index 62a09a9655c3..eb758aaa4374 100644
@@ -389,11 +389,15 @@ static int split_external_spt(struct kvm *kvm, gfn_t gfn, u64 old_spte,
 			      u64 new_spte, int level, bool shared)
 {
 	void *external_spt = get_external_spt(gfn, new_spte, level);
+	kvm_pfn_t pfn_for_gfn = spte_to_pfn(old_spte);
 	int ret;
 
 	KVM_BUG_ON(!external_spt, kvm);
 
-	ret = kvm_x86_call(split_external_spt)(kvm, gfn, level, external_spt, shared);
+	ret = kvm_x86_call(split_external_spt)(kvm, gfn, level,
+					       pfn_for_gfn, external_spt,
+					       shared);
+
 	return ret;
 }
 /**
arch/x86/kvm/vmx/tdx.c +2 -1
index 71115058e5e6..24aa9aaad6d8 100644
@@ -1941,7 +1941,8 @@ static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
 }
 
 static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,
-				      void *private_spt, bool mmu_lock_shared)
+				      kvm_pfn_t pfn_for_gfn, void *private_spt,
+				      bool mmu_lock_shared)
 {
 	struct page *page = virt_to_page(private_spt);
 	int ret;
Signature · 1 line
2.43.2
#112 lore raw

Re: [RFC PATCH v2 19/23] KVM: TDX: Pass down pfn to split_external_spt()

Quoted 39 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Pass down pfn to kvm_x86_ops::split_external_spt(). It is required for handling Dynamic PAMT in tdx_sept_split_private_spt().

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan)

---
  arch/x86/include/asm/kvm_host.h | 3 ++-
  arch/x86/kvm/mmu/tdp_mmu.c      | 6 +++++-
  arch/x86/kvm/vmx/tdx.c          | 3 ++-
  3 files changed, 9 insertions(+), 3 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -1
index 6cb5b422dd1d..6b6c46c27390 100644
@@ -1841,7 +1841,8 @@ struct kvm_x86_ops {
 
  	/* Split the external page table into smaller page tables */
  	int (*split_external_spt)(struct kvm *kvm, gfn_t gfn, enum pg_level level,
-				  void *external_spt, bool mmu_lock_shared);
+				  kvm_pfn_t pfn_for_gfn, void *external_spt,
+				  bool mmu_lock_shared);
 
  	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/tdp_mmu.c +0 -0
index 62a09a9655c3..eb758aaa4374 100644
@@ -389,11 +389,15 @@ static int split_external_spt(struct kvm *kvm, gfn_t gfn, u64 old_spte,
  			      u64 new_spte, int level, bool shared)
  {

arch/x86/kvm/mmu/tdp_mmu.c static int split_external_spt(struct kvm *kvm, gfn_t gfn, u64 old_spte, · line 399

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -1
  	void *external_spt = get_external_spt(gfn, new_spte, level);
+	kvm_pfn_t pfn_for_gfn = spte_to_pfn(old_spte);
  	int ret;
 
  	KVM_BUG_ON(!external_spt, kvm);
 
-	ret = kvm_x86_call(split_external_spt)(kvm, gfn, level, external_spt, shared);
+	ret = kvm_x86_call(split_external_spt)(kvm, gfn, level,
+					       pfn_for_gfn, external_spt,
+					       shared);

It can save one line by moving "pfn_for_gfn" up.

Quoted 18 lines
arch/x86/kvm/mmu/tdp_mmu.c +1 -0
+
  	return ret;
  }
  /**
arch/x86/kvm/vmx/tdx.c +2 -1
index 71115058e5e6..24aa9aaad6d8 100644
@@ -1941,7 +1941,8 @@ static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
  }
 
  static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level level,
-				      void *private_spt, bool mmu_lock_shared)
+				      kvm_pfn_t pfn_for_gfn, void *private_spt,
+				      bool mmu_lock_shared)
  {
  	struct page *page = virt_to_page(private_spt);
  	int ret;
author
#113 lore raw

RFC PATCH v2 20/23 KVM: TDX: Handle Dynamic PAMT in tdh_mem_page_demote()

In reply to Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

If Dynamic PAMT is enabled, TDH.MEM.PAGE.DEMOTE will take the PAMT page pair in registers R12 and R13.

Pass the pamt_pages list down to tdh_mem_page_demote() and populate registers R12 and R13 from it.

Instead of using seamcall_ret(), use seamcall_saved_ret() as it can handle registers above R11.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan).

---
 arch/x86/include/asm/tdx.h  |  1 +
 arch/x86/kvm/vmx/tdx.c      |  4 ++--
 arch/x86/virt/vmx/tdx/tdx.c | 13 +++++++++++--
 3 files changed, 14 insertions(+), 4 deletions(-)
arch/x86/include/asm/tdx.h +1 -0
index c058a82d4a97..2e529f0c578a 100644
@@ -180,6 +180,7 @@ u64 tdh_mng_create(struct tdx_td *td, u16 hkid);
 u64 tdh_vp_create(struct tdx_td *td, struct tdx_vp *vp);
 u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data);
 u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
+			struct list_head *pamt_pages,
 			u64 *ext_err1, u64 *ext_err2);
 u64 tdh_mr_extend(struct tdx_td *td, u64 gpa, u64 *ext_err1, u64 *ext_err2);
 u64 tdh_mr_finalize(struct tdx_td *td);
arch/x86/kvm/vmx/tdx.c +2 -2
index 24aa9aaad6d8..9d24a1a86a23 100644
@@ -1924,12 +1924,12 @@ static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
 	u64 err, entry, level_state;
 
 	err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
-				  &entry, &level_state);
+				  NULL, &entry, &level_state);
 
 	if (unlikely(tdx_operand_busy(err))) {
 		tdx_no_vcpus_enter_start(kvm);
 		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
-					  &entry, &level_state);
+					  NULL, &entry, &level_state);
 		tdx_no_vcpus_enter_stop(kvm);
 	}
 
arch/x86/virt/vmx/tdx/tdx.c +11 -2
index b7a0ee0f4a50..50f9d49f1c91 100644
@@ -1825,6 +1825,7 @@ u64 tdh_mng_rd(struct tdx_td *td, u64 field, u64 *data)
 EXPORT_SYMBOL_GPL(tdh_mng_rd);
 
 u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page,
+			struct list_head *pamt_pages,
 			u64 *ext_err1, u64 *ext_err2)
 {
 	struct tdx_module_args args = {
@@ -1832,10 +1833,18 @@ u64 tdh_mem_page_demote(struct tdx_td *td, u64 gpa, int level, struct page *page
 		.rdx = tdx_tdr_pa(td),
 		.r8 = page_to_phys(page),
 	};
-	u64 ret;
+	struct page *pamt_page;
+	u64 *p, ret;
 
+	if (level == TDX_PS_2M) {
+		p = &args.r12;
+		list_for_each_entry(pamt_page, pamt_pages, lru) {
+			*p = page_to_phys(pamt_page);
+			p++;
+		}
+	}
 	tdx_clflush_page(page);
-	ret = seamcall_ret(TDH_MEM_PAGE_DEMOTE, &args);
+	ret = seamcall_saved_ret(TDH_MEM_PAGE_DEMOTE, &args);
 
 	*ext_err1 = args.rcx;
 	*ext_err2 = args.rdx;
Signature · 1 line
2.43.2
author
#114 lore raw

RFC PATCH v2 21/23 KVM: TDX: Preallocate PAMT pages to be used in split path

In reply to Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Preallocate a page to be used in the split_external_spt() path.

Kernel needs one PAMT page pair for external_spt and one that provided directly to the TDH.MEM.PAGE.DEMOTE SEAMCALL.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Implemented the flow of topup pamt_page_cache in tdp_mmu_split_huge_pages_root() (Yan)

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu/mmu.c          |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      | 51 +++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+)
arch/x86/include/asm/kvm_host.h +2 -0
index 6b6c46c27390..508b133df903 100644
@@ -1591,6 +1591,8 @@ struct kvm_arch {
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
 	struct kvm_mmu_memory_cache split_desc_cache;
 
+	struct kvm_mmu_memory_cache pamt_page_cache;
+
 	gfn_t gfn_direct_bits;
 
 	/*
arch/x86/kvm/mmu/mmu.c +1 -0
index f23d8fc59323..e581cee37f64 100644
@@ -6848,6 +6848,7 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
 	kvm_mmu_free_memory_cache(&kvm->arch.split_desc_cache);
 	kvm_mmu_free_memory_cache(&kvm->arch.split_page_header_cache);
 	kvm_mmu_free_memory_cache(&kvm->arch.split_shadow_page_cache);
+	kvm_mmu_free_memory_cache(&kvm->arch.pamt_page_cache);
 }
 
 void kvm_mmu_uninit_vm(struct kvm *kvm)
arch/x86/kvm/mmu/tdp_mmu.c +51 -0

63 lines not on this page · raw message

Signature · 1 line
2.43.2
#115 lore raw

Re: [RFC PATCH v2 21/23] KVM: TDX: Preallocate PAMT pages to be used in split path

Quoted 3 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Preallocate a page to be used in the split_external_spt() path.

Not just "a" page.

Quoted 54 lines

Kernel needs one PAMT page pair for external_spt and one that provided directly to the TDH.MEM.PAGE.DEMOTE SEAMCALL.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Implemented the flow of topup pamt_page_cache in tdp_mmu_split_huge_pages_root() (Yan)

---
  arch/x86/include/asm/kvm_host.h |  2 ++
  arch/x86/kvm/mmu/mmu.c          |  1 +
  arch/x86/kvm/mmu/tdp_mmu.c      | 51 +++++++++++++++++++++++++++++++++
  3 files changed, 54 insertions(+)
arch/x86/include/asm/kvm_host.h +2 -0
index 6b6c46c27390..508b133df903 100644
@@ -1591,6 +1591,8 @@ struct kvm_arch {
  #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
  	struct kvm_mmu_memory_cache split_desc_cache;
 
+	struct kvm_mmu_memory_cache pamt_page_cache;
+
  	gfn_t gfn_direct_bits;
 
  	/*
arch/x86/kvm/mmu/mmu.c +1 -0
index f23d8fc59323..e581cee37f64 100644
@@ -6848,6 +6848,7 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
  	kvm_mmu_free_memory_cache(&kvm->arch.split_desc_cache);
  	kvm_mmu_free_memory_cache(&kvm->arch.split_page_header_cache);
  	kvm_mmu_free_memory_cache(&kvm->arch.split_shadow_page_cache);
+	kvm_mmu_free_memory_cache(&kvm->arch.pamt_page_cache);
  }
 
  void kvm_mmu_uninit_vm(struct kvm *kvm)
arch/x86/kvm/mmu/tdp_mmu.c +2 -0
index eb758aaa4374..064c4e823658 100644
@@ -1584,6 +1584,27 @@ static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end)
  		 (iter->gfn + KVM_PAGES_PER_HPAGE(iter->level)) <= end);
  }
 
+static bool need_topup_mirror_caches(struct kvm *kvm)
+{

arch/x86/kvm/mmu/tdp_mmu.c static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end) · line 1598

Quoted 10 lines
arch/x86/kvm/mmu/tdp_mmu.c +10 -0
+	int nr = tdx_nr_pamt_pages() * 2;
+
+	return kvm_mmu_memory_cache_nr_free_objects(&kvm->arch.pamt_page_cache) < nr;
+}
+
+static int topup_mirror_caches(struct kvm *kvm)
+{
+	int r, nr;
+
+	/* One for external_spt, one for TDH.MEM.PAGE.DEMOTE */

The comment is a bit confusing.
IIUC, external_spt is also for TDH.MEM.PAGE.DEMOTE. and it's "one pair" for PAMT pages.

arch/x86/kvm/mmu/tdp_mmu.c static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end) · line 1605

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +7 -0
+	nr = tdx_nr_pamt_pages() * 2;
+
+	r = kvm_mmu_topup_memory_cache(&kvm->arch.pamt_page_cache, nr);
+	if (r)
+		return r;
+
+	return 0;

This could be simplified:

arch/x86/kvm/mmu/tdp_mmu.c +0 -0
index 064c4e823658..35d052aa408c 100644
@@ -1593,16 +1593,12 @@ static bool need_topup_mirror_caches(struct kvm *kvm)
 

 static int topup_mirror_caches(struct kvm *kvm)
 {
-       int r, nr;
+       int nr;

        /* One for external_spt, one for TDH.MEM.PAGE.DEMOTE */ nr = tdx_nr_pamt_pages() * 2;

-       r = kvm_mmu_topup_memory_cache(&kvm->arch.pamt_page_cache, nr); -       if (r)
-               return r;
-
-       return 0;
+       return kvm_mmu_topup_memory_cache(&kvm->arch.pamt_page_cache, nr); }

Quoted 42 lines
arch/x86/kvm/mmu/tdp_mmu.c +32 -0
+}
+
  static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  					 struct kvm_mmu_page *root,
  					 gfn_t start, gfn_t end,
@@ -1656,6 +1677,36 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  			continue;
  		}
 
+		if (is_mirror_sp(root) && need_topup_mirror_caches(kvm)) {
+			int r;
+
+			rcu_read_unlock();
+
+			if (shared)
+				read_unlock(&kvm->mmu_lock);
+			else
+				write_unlock(&kvm->mmu_lock);
+
+			r = topup_mirror_caches(kvm);
+
+			if (shared)
+				read_lock(&kvm->mmu_lock);
+			else
+				write_lock(&kvm->mmu_lock);
+
+			if (r) {
+				trace_kvm_mmu_split_huge_page(iter.gfn,
+							      iter.old_spte,
+							      iter.level, r);
+				return r;
+			}
+
+			rcu_read_lock();
+
+			iter.yielded = true;
+			continue;
+		}
+
  		tdp_mmu_init_child_sp(sp, &iter);
 
  		if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared))
author
#116 lore raw

Re: [RFC PATCH v2 21/23] KVM: TDX: Preallocate PAMT pages to be used in split path

Quoted 65 lines · Binbin Wu
Quoted 3 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Preallocate a page to be used in the split_external_spt() path.

Not just "a" page.

Quoted 56 lines

Kernel needs one PAMT page pair for external_spt and one that provided directly to the TDH.MEM.PAGE.DEMOTE SEAMCALL.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Implemented the flow of topup pamt_page_cache in tdp_mmu_split_huge_pages_root() (Yan)

---
  arch/x86/include/asm/kvm_host.h |  2 ++
  arch/x86/kvm/mmu/mmu.c          |  1 +
  arch/x86/kvm/mmu/tdp_mmu.c      | 51 +++++++++++++++++++++++++++++++++
  3 files changed, 54 insertions(+)
arch/x86/include/asm/kvm_host.h +2 -0
index 6b6c46c27390..508b133df903 100644
@@ -1591,6 +1591,8 @@ struct kvm_arch {
  #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
  	struct kvm_mmu_memory_cache split_desc_cache;
+	struct kvm_mmu_memory_cache pamt_page_cache;
+
  	gfn_t gfn_direct_bits;
  	/*
arch/x86/kvm/mmu/mmu.c +1 -0
index f23d8fc59323..e581cee37f64 100644
@@ -6848,6 +6848,7 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
  	kvm_mmu_free_memory_cache(&kvm->arch.split_desc_cache);
  	kvm_mmu_free_memory_cache(&kvm->arch.split_page_header_cache);
  	kvm_mmu_free_memory_cache(&kvm->arch.split_shadow_page_cache);
+	kvm_mmu_free_memory_cache(&kvm->arch.pamt_page_cache);
  }
  void kvm_mmu_uninit_vm(struct kvm *kvm)
arch/x86/kvm/mmu/tdp_mmu.c +8 -0
index eb758aaa4374..064c4e823658 100644
@@ -1584,6 +1584,27 @@ static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end)
  		 (iter->gfn + KVM_PAGES_PER_HPAGE(iter->level)) <= end);
  }
+static bool need_topup_mirror_caches(struct kvm *kvm)
+{
+	int nr = tdx_nr_pamt_pages() * 2;
+
+	return kvm_mmu_memory_cache_nr_free_objects(&kvm->arch.pamt_page_cache) < nr;
+}
+
+static int topup_mirror_caches(struct kvm *kvm)
Quoted 8 lines

arch/x86/kvm/mmu/tdp_mmu.c static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end) · line 1597

Quoted 4 lines
arch/x86/kvm/mmu/tdp_mmu.c +4 -0
+{
+	int r, nr;
+
+	/* One for external_spt, one for TDH.MEM.PAGE.DEMOTE */

The comment is a bit confusing.
IIUC, external_spt is also for TDH.MEM.PAGE.DEMOTE. and it's "one pair" for PAMT pages.

Sould be
one pair of PAMT pages for the adding page table page used by splitting, and another pair for guest private page to be demoted.

Quoted 9 lines

arch/x86/kvm/mmu/tdp_mmu.c static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end) · line 1604

Quoted 7 lines
arch/x86/kvm/mmu/tdp_mmu.c +7 -0
+	nr = tdx_nr_pamt_pages() * 2;
+
+	r = kvm_mmu_topup_memory_cache(&kvm->arch.pamt_page_cache, nr);
+	if (r)
+		return r;
+
+	return 0;

This could be simplified:

Indeed.

Quoted 63 lines
arch/x86/kvm/mmu/tdp_mmu.c +2 -6
index 064c4e823658..35d052aa408c 100644
@@ -1593,16 +1593,12 @@ static bool need_topup_mirror_caches(struct kvm *kvm)
 
  static int topup_mirror_caches(struct kvm *kvm)
  {
-       int r, nr;
+       int nr;
 
         /* One for external_spt, one for TDH.MEM.PAGE.DEMOTE */
         nr = tdx_nr_pamt_pages() * 2;
 
-       r = kvm_mmu_topup_memory_cache(&kvm->arch.pamt_page_cache, nr);
-       if (r)
-               return r;
-
-       return 0;
+       return kvm_mmu_topup_memory_cache(&kvm->arch.pamt_page_cache, nr);
  }
 
 > +}
Quoted 39 lines
arch/x86/kvm/mmu/tdp_mmu.c +31 -0
+
  static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  					 struct kvm_mmu_page *root,
  					 gfn_t start, gfn_t end,
@@ -1656,6 +1677,36 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
  			continue;
  		}
+		if (is_mirror_sp(root) && need_topup_mirror_caches(kvm)) {
+			int r;
+
+			rcu_read_unlock();
+
+			if (shared)
+				read_unlock(&kvm->mmu_lock);
+			else
+				write_unlock(&kvm->mmu_lock);
+
+			r = topup_mirror_caches(kvm);
+
+			if (shared)
+				read_lock(&kvm->mmu_lock);
+			else
+				write_lock(&kvm->mmu_lock);
+
+			if (r) {
+				trace_kvm_mmu_split_huge_page(iter.gfn,
+							      iter.old_spte,
+							      iter.level, r);
+				return r;
+			}
+
+			rcu_read_lock();
+
+			iter.yielded = true;
+			continue;
+		}
+
  		tdp_mmu_init_child_sp(sp, &iter);
  		if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared))
#117 lore raw

Re: [RFC PATCH v2 21/23] KVM: TDX: Preallocate PAMT pages to be used in split path

In reply to Yan Zhao
Quoted 28 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Preallocate a page to be used in the split_external_spt() path.

Kernel needs one PAMT page pair for external_spt and one that provided directly to the TDH.MEM.PAGE.DEMOTE SEAMCALL.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Implemented the flow of topup pamt_page_cache in tdp_mmu_split_huge_pages_root() (Yan)

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu/mmu.c          |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      | 51 +++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+)
arch/x86/include/asm/kvm_host.h +0 -0
index 6b6c46c27390..508b133df903 100644

arch/x86/include/asm/kvm_host.h struct kvm_arch { · line 1595

Quoted 5 lines
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1591,6 +1591,8 @@ struct kvm_arch {
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
        struct kvm_mmu_memory_cache split_desc_cache;
 
+       struct kvm_mmu_memory_cache pamt_page_cache;
+

The latest DPAMT patches use a per-vcpu tdx_prealloc struct to handle preallocating pages for pamt. I'm wondering if you've considered how this would work here since some of the calls requiring pamt originate from user space ioctls and therefore are not associated with a vcpu.

Since the tdx_prealloc is a per vcpu struct there are no race issues when multiple vcpus need to add pamt pages but here it would be trickier here because theoretically, multiple threads could split different pages simultaneously.

Quoted 88 lines
arch/x86/include/asm/kvm_host.h +0 -0
        gfn_t gfn_direct_bits;
 
        /*
arch/x86/kvm/mmu/mmu.c +1 -0
index f23d8fc59323..e581cee37f64 100644
@@ -6848,6 +6848,7 @@ static void mmu_free_vm_memory_caches(struct kvm *kvm)
        kvm_mmu_free_memory_cache(&kvm->arch.split_desc_cache);
        kvm_mmu_free_memory_cache(&kvm->arch.split_page_header_cache);
        kvm_mmu_free_memory_cache(&kvm->arch.split_shadow_page_cache);
+       kvm_mmu_free_memory_cache(&kvm->arch.pamt_page_cache);
 }
 
 void kvm_mmu_uninit_vm(struct kvm *kvm)
arch/x86/kvm/mmu/tdp_mmu.c +51 -1
index eb758aaa4374..064c4e823658 100644
@@ -1584,6 +1584,27 @@ static bool iter_cross_boundary(struct tdp_iter *iter, gfn_t start, gfn_t end)
                 (iter->gfn + KVM_PAGES_PER_HPAGE(iter->level)) <= end);
 }
 
+static bool need_topup_mirror_caches(struct kvm *kvm)
+{
+       int nr = tdx_nr_pamt_pages() * 2;
+
+       return kvm_mmu_memory_cache_nr_free_objects(&kvm->arch.pamt_page_cache) < nr;
+}
+
+static int topup_mirror_caches(struct kvm *kvm)
+{
+       int r, nr;
+
+       /* One for external_spt, one for TDH.MEM.PAGE.DEMOTE */
+       nr = tdx_nr_pamt_pages() * 2;
+
+       r = kvm_mmu_topup_memory_cache(&kvm->arch.pamt_page_cache, nr);
+       if (r)
+               return r;
+
+       return 0;
+}
+
 static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                                         struct kvm_mmu_page *root,
                                         gfn_t start, gfn_t end,
@@ -1656,6 +1677,36 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
                        continue;
                }
 
+               if (is_mirror_sp(root) && need_topup_mirror_caches(kvm)) {
+                       int r;
+
+                       rcu_read_unlock();
+
+                       if (shared)
+                               read_unlock(&kvm->mmu_lock);
+                       else
+                               write_unlock(&kvm->mmu_lock);
+
+                       r = topup_mirror_caches(kvm);
+
+                       if (shared)
+                               read_lock(&kvm->mmu_lock);
+                       else
+                               write_lock(&kvm->mmu_lock);
+
+                       if (r) {
+                               trace_kvm_mmu_split_huge_page(iter.gfn,
+                                                             iter.old_spte,
+                                                             iter.level, r);
+                               return r;
+                       }
+
+                       rcu_read_lock();
+
+                       iter.yielded = true;
+                       continue;
+               }
+
                tdp_mmu_init_child_sp(sp, &iter);
 
                if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared))
--

2.43.2

author
#118 lore raw

Re: [RFC PATCH v2 21/23] KVM: TDX: Preallocate PAMT pages to be used in split path

Quoted 32 lines · Sagi Shahar
Quoted 31 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Preallocate a page to be used in the split_external_spt() path.

Kernel needs one PAMT page pair for external_spt and one that provided directly to the TDH.MEM.PAGE.DEMOTE SEAMCALL.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Implemented the flow of topup pamt_page_cache in tdp_mmu_split_huge_pages_root() (Yan)

---
 arch/x86/include/asm/kvm_host.h |  2 ++
 arch/x86/kvm/mmu/mmu.c          |  1 +
 arch/x86/kvm/mmu/tdp_mmu.c      | 51 +++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+)
arch/x86/include/asm/kvm_host.h +0 -0
index 6b6c46c27390..508b133df903 100644
@@ -1591,6 +1591,8 @@ struct kvm_arch {
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
        struct kvm_mmu_memory_cache split_desc_cache;
 
Quoted 7 lines

arch/x86/include/asm/kvm_host.h struct kvm_arch { · line 1595

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +2 -0
+       struct kvm_mmu_memory_cache pamt_page_cache;
+

The latest DPAMT patches use a per-vcpu tdx_prealloc struct to handle preallocating pages for pamt. I'm wondering if you've considered how this would work here since some of the calls requiring pamt originate from user space ioctls and therefore are not associated with a vcpu.

I'll use a per-VM tdx_prealloc struct for splitting here, similar to the per-VM pamt_page_cache.

arch/x86/kvm/vmx/tdx.h +3 -0
index 43dd295b7fd6..91bea25da528 100644
@@ -48,6 +48,9 @@ struct kvm_tdx {
         * Set/unset is protected with kvm->mmu_lock.
         */
        bool wait_for_sept_zap;
+
+       spinlock_t external_kvm_split_cache_lock;
+       struct tdx_prealloc prealloc_split_cache;
 };
 
Quoted 4 lines

Since the tdx_prealloc is a per vcpu struct there are no race issues when multiple vcpus need to add pamt pages but here it would be trickier here because theoretically, multiple threads could split different pages simultaneously.

A spin lock external_kvm_split_cache_lock is introduced to protect the cache enqueue and dequeue.
(a) When tdp_mmu_split_huge_pages_root() is invoked under write mmu_lock:
- Since cache dequeue is already under write mmu_lock in tdp_mmu_split_huge_page()-->tdx_sept_split_private_spte(), acquiring/ releasing another spin lock doesn't matter.
- Though the cache enqueue in topup_external_split_cache() is not protected by mmu_lock, protecting enqueue with a spinlock should not reduce concurrency.

(b) When tdp_mmu_split_huge_pages_root() is invoked under read mmu_lock:
Introducing a new spinlock may hurt concurrency for a brief duration (which is necessary).
However, there's no known (future) use case for multiple threads invoking tdp_mmu_split_huge_pages_root() on mirror root under shared mmu_lock.

For future splitting under shared mmu_lock in the fault path, we'll use the per-vCPU tdx_prealloc instead of the per-VM cache. TDX can leverage kvm_get_running_vcpu() to differentiate between the two caches.

Here's the new diff in TDP MMU rebased to Dynamic PAMT v4.

+/*
+ * Check the per-VM external split cache under write mmu_lock or read mmu_lock
+ * in tdp_mmu_split_huge_pages_root().
+ *
+ * When need_topup_external_split_cache() returns true, the mmu_lock is held
+ * throughout
+ * (a) need_topup_external_split_cache(), and
+ * (b) the cache consumption (in tdx_sept_split_private_spte() called by
+ * tdp_mmu_split_huge_page()).
+ *
+ * Throughout the execution from (a) to (b):
+ * - With write mmu_lock, the per-VM external split cache is exclusively
+ * accessed by a single user. Therefore, the result returned from
+ * need_topup_external_split_cache() is accurate.
+ *
+ * - With read mmu_lock, the per-VM external split cache can be shared among

+ *   multiple users. Cache consumption in tdx_sept_split_private_spte() thus
+ *   needs to check again of the cache page count after acquiring its internal
+ *   split cache lock and return an error if the cache page count is not
+ *   sufficient.

+ */
+static bool need_topup_external_split_cache(struct kvm *kvm, int level) +{
+ return kvm_x86_call(need_topup_external_per_vm_split_cache)(kvm, level);
+}
+static int topup_external_split_cache(struct kvm *kvm, int level, bool shared) +{
+ int r;
+
+ rcu_read_unlock();
+
+ if (shared)
+ read_unlock(&kvm->mmu_lock);
+ else
+ write_unlock(&kvm->mmu_lock);
+
+ r = kvm_x86_call(topup_external_per_vm_split_cache)(kvm, level);
+
+ if (shared)
+ read_lock(&kvm->mmu_lock);
+ else
+ write_lock(&kvm->mmu_lock);
+
+ if (!r)
+ rcu_read_lock();
+
+ return r;
+}
static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,
struct kvm_mmu_page *root, gfn_t start, gfn_t end,
@@ -1673,6 +1723,23 @@ static int tdp_mmu_split_huge_pages_root(struct kvm *kvm,

                        continue;
                }

+ if (is_mirror_sp(root) &&
+ need_topup_external_split_cache(kvm, iter.level)) {
+ int r;
+
+ r = topup_external_split_cache(kvm, iter.level, shared);
+
+ if (r) {
+ trace_kvm_mmu_split_huge_page(iter.gfn,
+ iter.old_spte,
+ iter.level, r);
+ return r;
+ }
+
+ iter.yielded = true;
+ continue;
+ }
+
tdp_mmu_init_child_sp(sp, &iter);

if (tdp_mmu_split_huge_page(kvm, &iter, sp, shared))

#119 lore raw

Re: [RFC PATCH v2 21/23] KVM: TDX: Preallocate PAMT pages to be used in split path

Quoted 36 lines · Yan Zhao
Quoted 9 lines
Quoted 9 lines
arch/x86/include/asm/kvm_host.h +0 -0
index 6b6c46c27390..508b133df903 100644
@@ -1591,6 +1591,8 @@ struct kvm_arch {
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
        struct kvm_mmu_memory_cache split_desc_cache;
 
Quoted 7 lines

arch/x86/include/asm/kvm_host.h struct kvm_arch { · line 1595

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +2 -0
+       struct kvm_mmu_memory_cache pamt_page_cache;
+

The latest DPAMT patches use a per-vcpu tdx_prealloc struct to handle preallocating pages for pamt. I'm wondering if you've considered how this would work here since some of the calls requiring pamt originate from user space ioctls and therefore are not associated with a vcpu.

I'll use a per-VM tdx_prealloc struct for splitting here, similar to the per-VM pamt_page_cache.

arch/x86/kvm/vmx/tdx.h +3 -0
index 43dd295b7fd6..91bea25da528 100644
@@ -48,6 +48,9 @@ struct kvm_tdx {
         * Set/unset is protected with kvm->mmu_lock.
         */
        bool wait_for_sept_zap;
+
+       spinlock_t external_kvm_split_cache_lock;
+       struct tdx_prealloc prealloc_split_cache;
 };
 
Quoted 4 lines

Since the tdx_prealloc is a per vcpu struct there are no race issues when multiple vcpus need to add pamt pages but here it would be trickier here because theoretically, multiple threads could split different pages simultaneously.

Quoted 9 lines

A spin lock external_kvm_split_cache_lock is introduced to protect the cache enqueue and dequeue.
(a) When tdp_mmu_split_huge_pages_root() is invoked under write mmu_lock:
- Since cache dequeue is already under write mmu_lock in tdp_mmu_split_huge_page()-->tdx_sept_split_private_spte(), acquiring/ releasing another spin lock doesn't matter.
- Though the cache enqueue in topup_external_split_cache() is not protected by mmu_lock, protecting enqueue with a spinlock should not reduce concurrency.

Even with the spin lock protecting the cache topup/consumption operation, is it possible that one split operation context consumes the top-up performed by the other split operation causing failure with the subsequent consumptions?

Quoted 12 lines

(b) When tdp_mmu_split_huge_pages_root() is invoked under read mmu_lock:
Introducing a new spinlock may hurt concurrency for a brief duration (which is necessary).
However, there's no known (future) use case for multiple threads invoking tdp_mmu_split_huge_pages_root() on mirror root under shared mmu_lock.

For future splitting under shared mmu_lock in the fault path, we'll use the per-vCPU tdx_prealloc instead of the per-VM cache. TDX can leverage kvm_get_running_vcpu() to differentiate between the two caches.

author
#120 lore raw

Re: [RFC PATCH v2 21/23] KVM: TDX: Preallocate PAMT pages to be used in split path

Quoted 47 lines · Vishal Annapurve
Quoted 36 lines · Yan Zhao
Quoted 9 lines
Quoted 9 lines
arch/x86/include/asm/kvm_host.h +0 -0
index 6b6c46c27390..508b133df903 100644
@@ -1591,6 +1591,8 @@ struct kvm_arch {
 #define SPLIT_DESC_CACHE_MIN_NR_OBJECTS (SPTE_ENT_PER_PAGE + 1)
        struct kvm_mmu_memory_cache split_desc_cache;
 
Quoted 7 lines

arch/x86/include/asm/kvm_host.h struct kvm_arch { · line 1595

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +2 -0
+       struct kvm_mmu_memory_cache pamt_page_cache;
+

The latest DPAMT patches use a per-vcpu tdx_prealloc struct to handle preallocating pages for pamt. I'm wondering if you've considered how this would work here since some of the calls requiring pamt originate from user space ioctls and therefore are not associated with a vcpu.

I'll use a per-VM tdx_prealloc struct for splitting here, similar to the per-VM pamt_page_cache.

arch/x86/kvm/vmx/tdx.h +3 -0
index 43dd295b7fd6..91bea25da528 100644
@@ -48,6 +48,9 @@ struct kvm_tdx {
         * Set/unset is protected with kvm->mmu_lock.
         */
        bool wait_for_sept_zap;
+
+       spinlock_t external_kvm_split_cache_lock;
+       struct tdx_prealloc prealloc_split_cache;
 };
 
Quoted 4 lines

Since the tdx_prealloc is a per vcpu struct there are no race issues when multiple vcpus need to add pamt pages but here it would be trickier here because theoretically, multiple threads could split different pages simultaneously.

Quoted 9 lines

A spin lock external_kvm_split_cache_lock is introduced to protect the cache enqueue and dequeue.
(a) When tdp_mmu_split_huge_pages_root() is invoked under write mmu_lock:
- Since cache dequeue is already under write mmu_lock in tdp_mmu_split_huge_page()-->tdx_sept_split_private_spte(), acquiring/ releasing another spin lock doesn't matter.
- Though the cache enqueue in topup_external_split_cache() is not protected by mmu_lock, protecting enqueue with a spinlock should not reduce concurrency.

Quoted 4 lines

Even with the spin lock protecting the cache topup/consumption operation, is it possible that one split operation context consumes the top-up performed by the other split operation causing failure with the subsequent consumptions?

The sequence of check topup, topup, and consume is like this

1. write_lock(&kvm->mmu_lock)
check topup

2. write_unlock(&kvm->mmu_lock)
topup (get/put split lock to enqueue)

3. write_lock(&kvm->mmu_lock)
check topup (goto 2 if topup is necessary) (*) get split lock
consume
put split lock write_unlock(&kvm->mmu_lock)

Note: due to the "iter.yielded = true" and "continue" after the topup, (see my posted diff in last reply), consuming does not directly follow the topup. i.e., there is step 3.

Due to (*) in step 3, and the consuming is under write mmu_lock, it's impossible for splits in other threads to consume pages allocated for this split.

Quoted 3 lines
Quoted 3 lines

(b) When tdp_mmu_split_huge_pages_root() is invoked under read mmu_lock:
Introducing a new spinlock may hurt concurrency for a brief duration (which is necessary).

Let's talk more about this future potential use cases (i.e., if there're multiple callers of tdp_mmu_split_huge_pages_root() under shared mmu_lock). The sequence would be

1. read_lock(&kvm->mmu_lock)
check topup

2. read_unlock(&kvm->mmu_lock)
topup (get/put split lock to enqueue)

3. read_lock(&kvm->mmu_lock)
check topup (goto 2 if topup is necessary)

get split lock
check topup (return retry if topup is necessary) (**) consume
put split lock
read_unlock(&kvm->mmu_lock)

Due to (**) in step 3, and the consuming is under split lock and read mmu_lock, it's also impossible for splits in other threads to consume pages allocated for this split.

Quoted 8 lines
Quoted 8 lines

However, there's no known (future) use case for multiple threads invoking tdp_mmu_split_huge_pages_root() on mirror root under shared mmu_lock.

For future splitting under shared mmu_lock in the fault path, we'll use the per-vCPU tdx_prealloc instead of the per-VM cache. TDX can leverage kvm_get_running_vcpu() to differentiate between the two caches.

author
#121 lore raw

RFC PATCH v2 22/23 KVM: TDX: Handle Dynamic PAMT on page split

In reply to Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

Page demote from 2M to 4k requires an additional PAMT page pair to cover the 2M range that now mapped with 4k.

EPT page also has to be covered in PAMT_4K.

Allocate both from pre-allocated split PAMT pool.

---
RFC v2:
- Pulled from
git://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git tdx/dpamt-huge.
- Rebased on top of TDX huge page RFC v2 (Yan).

---
 arch/x86/include/asm/tdx.h  |  4 ++++
 arch/x86/kvm/vmx/tdx.c      | 28 ++++++++++++++++++++++++----
 arch/x86/virt/vmx/tdx/tdx.c | 11 +++++++----
 3 files changed, 35 insertions(+), 8 deletions(-)
arch/x86/include/asm/tdx.h +4 -0
index 2e529f0c578a..da317981e95a 100644
@@ -123,6 +123,10 @@ u32 tdx_get_nr_guest_keyids(void);
 void tdx_guest_keyid_free(unsigned int keyid);
 
 int tdx_nr_pamt_pages(void);
+atomic_t *tdx_get_pamt_refcount(unsigned long hpa);
+int tdx_alloc_pamt_pages(struct list_head *pamt_pages,
+			 struct page *(alloc)(void *data), void *data);
+void tdx_free_pamt_pages(struct list_head *pamt_pages);
 int tdx_pamt_get(struct page *page, enum pg_level level,
 		 struct page *(alloc)(void *data), void *data);
 void tdx_pamt_put(struct page *page, enum pg_level level);
arch/x86/kvm/vmx/tdx.c +24 -4
index 9d24a1a86a23..6e061d659639 100644
@@ -1915,28 +1915,48 @@ static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn,
 	return 0;
 }
 
+static struct page *tdx_alloc_pamt_page_split(void *data)
+{
+	struct kvm *kvm = data;
+	void *p;
+
+	p = kvm_mmu_memory_cache_alloc(&kvm->arch.pamt_page_cache);
+	return virt_to_page(p);
+}
+
 static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
-					enum pg_level level, struct page *page)
+					enum pg_level level, struct page *page,
+					kvm_pfn_t pfn_for_gfn)
 {
 	int tdx_level = pg_level_to_tdx_sept_level(level);
+	hpa_t hpa = pfn_to_hpa(pfn_for_gfn);
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
 	gpa_t gpa = gfn_to_gpa(gfn);
 	u64 err, entry, level_state;
+	LIST_HEAD(pamt_pages);
+
+	tdx_pamt_get(page, PG_LEVEL_4K, tdx_alloc_pamt_page_split, kvm);
+	tdx_alloc_pamt_pages(&pamt_pages, tdx_alloc_pamt_page_split, kvm);
 
 	err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
-				  NULL, &entry, &level_state);
+				  &pamt_pages, &entry, &level_state);
 
 	if (unlikely(tdx_operand_busy(err))) {
 		tdx_no_vcpus_enter_start(kvm);
 		err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
-					  NULL, &entry, &level_state);
+					  &pamt_pages, &entry, &level_state);
 		tdx_no_vcpus_enter_stop(kvm);
 	}
 
 	if (KVM_BUG_ON(err, kvm)) {
+		tdx_free_pamt_pages(&pamt_pages);
+		tdx_pamt_put(page, PG_LEVEL_4K);
 		pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
 		return -EIO;
 	}
+
+	if (tdx_supports_dynamic_pamt(tdx_sysinfo))
+		atomic_set(tdx_get_pamt_refcount(hpa), PTRS_PER_PMD);
 	return 0;
 }
 
@@ -1963,7 +1983,7 @@ static int tdx_sept_split_private_spt(struct kvm *kvm, gfn_t gfn, enum pg_level
 
 	tdx_track(kvm);
 
-	return tdx_spte_demote_private_spte(kvm, gfn, level, page);
+	return tdx_spte_demote_private_spte(kvm, gfn, level, page, pfn_for_gfn);
 }
 
 static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
arch/x86/virt/vmx/tdx/tdx.c +7 -4
index 50f9d49f1c91..dbbddd00ec60 100644
@@ -188,10 +188,11 @@ int tdx_cpu_enable(void)
 }
 EXPORT_SYMBOL_GPL(tdx_cpu_enable);
 
-static atomic_t *tdx_get_pamt_refcount(unsigned long hpa)
+atomic_t *tdx_get_pamt_refcount(unsigned long hpa)
 {
 	return &pamt_refcounts[hpa / PMD_SIZE];
 }
+EXPORT_SYMBOL_GPL(tdx_get_pamt_refcount);
 
 static int pamt_refcount_populate(pte_t *pte, unsigned long addr, void *data)
 {
@@ -2151,7 +2152,7 @@ static u64 tdh_phymem_pamt_remove(unsigned long hpa,
 
 static DEFINE_SPINLOCK(pamt_lock);
 
-static void tdx_free_pamt_pages(struct list_head *pamt_pages)
+void tdx_free_pamt_pages(struct list_head *pamt_pages)
 {
 	struct page *page;
 
@@ -2160,9 +2161,10 @@ static void tdx_free_pamt_pages(struct list_head *pamt_pages)
 		__free_page(page);
 	}
 }
+EXPORT_SYMBOL_GPL(tdx_free_pamt_pages);
 
-static int tdx_alloc_pamt_pages(struct list_head *pamt_pages,
-				 struct page *(alloc)(void *data), void *data)
+int tdx_alloc_pamt_pages(struct list_head *pamt_pages,
+			 struct page *(alloc)(void *data), void *data)
 {
 	for (int i = 0; i < tdx_nr_pamt_pages(); i++) {
 		struct page *page;
@@ -2180,6 +2182,7 @@ static int tdx_alloc_pamt_pages(struct list_head *pamt_pages,
 	tdx_free_pamt_pages(pamt_pages);
 	return -ENOMEM;
 }
+EXPORT_SYMBOL_GPL(tdx_alloc_pamt_pages);
 
 static int tdx_pamt_add(atomic_t *pamt_refcount, unsigned long hpa,
 			struct list_head *pamt_pages)
Signature · 1 line
2.43.2
#122 lore raw

Re: [RFC PATCH v2 22/23] KVM: TDX: Handle Dynamic PAMT on page split

Quoted 14 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

arch/x86/kvm/vmx/tdx.c +10 -1
+static struct page *tdx_alloc_pamt_page_split(void *data)
+{
+       struct kvm *kvm = data;
+       void *p;
+
+       p = kvm_mmu_memory_cache_alloc(&kvm->arch.pamt_page_cache);
+       return virt_to_page(p);
+}
+
 static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
-                                       enum pg_level level, struct page *page)
+                                       enum pg_level level, struct page *page,

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1938

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +5 -0
+                                       kvm_pfn_t pfn_for_gfn)
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
+       hpa_t hpa = pfn_to_hpa(pfn_for_gfn);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
+       LIST_HEAD(pamt_pages);
+
+       tdx_pamt_get(page, PG_LEVEL_4K, tdx_alloc_pamt_page_split, kvm);

This invocation needs a return value check.

Quoted 1 line

+ tdx_alloc_pamt_pages(&pamt_pages, tdx_alloc_pamt_page_split, kvm);

IIUC tdx_pamt_get() will result in pamt_pages allocation above, so this step is not needed.

Quoted 5 lines
        err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
arch/x86/kvm/vmx/tdx.c +1 -1
-                                 NULL, &entry, &level_state);
+                                 &pamt_pages, &entry, &level_state);
 

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1952

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +2 -1
        if (unlikely(tdx_operand_busy(err))) {
                tdx_no_vcpus_enter_start(kvm);
                err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
-                                         NULL, &entry, &level_state);
+                                         &pamt_pages, &entry, &level_state);
                tdx_no_vcpus_enter_stop(kvm);
        }
 
        if (KVM_BUG_ON(err, kvm)) {
+               tdx_free_pamt_pages(&pamt_pages);

If tdx_alloc_pamt_pages() is not needed then this can be dropped as well.

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1959

Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+               tdx_pamt_put(page, PG_LEVEL_4K);
                pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
                return -EIO;
        }
+
+       if (tdx_supports_dynamic_pamt(tdx_sysinfo))
+               atomic_set(tdx_get_pamt_refcount(hpa), PTRS_PER_PMD);

Should this be atomic_set(tdx_get_pamt_refcount(hpa), PTRS_PER_PMD -1 );

as tdx_pamt_get would have increased the refcount by 1 already above?

#123 lore raw

Re: [RFC PATCH v2 22/23] KVM: TDX: Handle Dynamic PAMT on page split

Quoted 29 lines · Vishal Annapurve
Quoted 14 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

arch/x86/kvm/vmx/tdx.c +10 -1
+static struct page *tdx_alloc_pamt_page_split(void *data)
+{
+       struct kvm *kvm = data;
+       void *p;
+
+       p = kvm_mmu_memory_cache_alloc(&kvm->arch.pamt_page_cache);
+       return virt_to_page(p);
+}
+
 static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
-                                       enum pg_level level, struct page *page)
+                                       enum pg_level level, struct page *page,

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1938

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +5 -0
+                                       kvm_pfn_t pfn_for_gfn)
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
+       hpa_t hpa = pfn_to_hpa(pfn_for_gfn);
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
+       LIST_HEAD(pamt_pages);
+
+       tdx_pamt_get(page, PG_LEVEL_4K, tdx_alloc_pamt_page_split, kvm);

This invocation needs a return value check.

Quoted 4 lines
Quoted 1 line

+ tdx_alloc_pamt_pages(&pamt_pages, tdx_alloc_pamt_page_split, kvm);

IIUC tdx_pamt_get() will result in pamt_pages allocation above, so this step is not needed.

I missed that one allocation is to cover the EPT page and another is for HPA ranges backing the GPA mappings. So ignore my rest of the comments except about the error handling for tdx_pamt_get() and tdx_alloc_pamt_pages() missing in this patch.

author
#124 lore raw

Re: [RFC PATCH v2 22/23] KVM: TDX: Handle Dynamic PAMT on page split

In reply to Vishal Annapurve
Quoted 19 lines · Vishal Annapurve
Quoted 18 lines · Yan Zhao

From: "Kirill A. Shutemov" <[email protected]>

arch/x86/kvm/vmx/tdx.c +12 -1
+static struct page *tdx_alloc_pamt_page_split(void *data)
+{
+       struct kvm *kvm = data;
+       void *p;
+
+       p = kvm_mmu_memory_cache_alloc(&kvm->arch.pamt_page_cache);
+       return virt_to_page(p);
+}
+
 static int tdx_spte_demote_private_spte(struct kvm *kvm, gfn_t gfn,
-                                       enum pg_level level, struct page *page)
+                                       enum pg_level level, struct page *page,
+                                       kvm_pfn_t pfn_for_gfn)
 {
        int tdx_level = pg_level_to_tdx_sept_level(level);
+       hpa_t hpa = pfn_to_hpa(pfn_for_gfn);
Quoted 8 lines

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1938

Quoted 6 lines
        struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
        gpa_t gpa = gfn_to_gpa(gfn);
        u64 err, entry, level_state;
arch/x86/kvm/vmx/tdx.c +3 -0
+       LIST_HEAD(pamt_pages);
+
+       tdx_pamt_get(page, PG_LEVEL_4K, tdx_alloc_pamt_page_split, kvm);

This invocation needs a return value check.

Ack.

Quoted 4 lines
Quoted 1 line

+ tdx_alloc_pamt_pages(&pamt_pages, tdx_alloc_pamt_page_split, kvm);

IIUC tdx_pamt_get() will result in pamt_pages allocation above, so this step is not needed.

This step is to allocate pamt_pages for the guest 2MB page that needs splitting. The above tdx_pamt_get() is for the EPT page to be added.
I'll add comments or update the param names for better clarity.

Regarding the absence of return value check for the tdx_alloc_pamt_pages(), I think it's because the tdx_alloc_pamt_page_split() retrieves pages from the pamt_page_cache via kvm_mmu_memory_cache_alloc(), which is guaranteed to succeed (otherwise, there's a BUG_ON() in kvm_mmu_memory_cache_alloc()).

Quoted 26 lines
Quoted 5 lines
        err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
arch/x86/kvm/vmx/tdx.c +1 -1
-                                 NULL, &entry, &level_state);
+                                 &pamt_pages, &entry, &level_state);
 

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1952

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +2 -1
        if (unlikely(tdx_operand_busy(err))) {
                tdx_no_vcpus_enter_start(kvm);
                err = tdh_mem_page_demote(&kvm_tdx->td, gpa, tdx_level, page,
-                                         NULL, &entry, &level_state);
+                                         &pamt_pages, &entry, &level_state);
                tdx_no_vcpus_enter_stop(kvm);
        }
 
        if (KVM_BUG_ON(err, kvm)) {
+               tdx_free_pamt_pages(&pamt_pages);

If tdx_alloc_pamt_pages() is not needed then this can be dropped as well.

arch/x86/kvm/vmx/tdx.c static int tdx_sept_free_private_spt(struct kvm *kvm, gfn_t gfn, · line 1959

Quoted 7 lines
arch/x86/kvm/vmx/tdx.c +4 -0
+               tdx_pamt_put(page, PG_LEVEL_4K);
                pr_tdx_error_2(TDH_MEM_PAGE_DEMOTE, err, entry, level_state);
                return -EIO;
        }
+
+       if (tdx_supports_dynamic_pamt(tdx_sysinfo))
+               atomic_set(tdx_get_pamt_refcount(hpa), PTRS_PER_PMD);
Quoted 4 lines

Should this be atomic_set(tdx_get_pamt_refcount(hpa), PTRS_PER_PMD -1 );

as tdx_pamt_get would have increased the refcount by 1 already above?

This hpa is for guest 2MB memory range. There shouldn't have any increased pamt_refcount for this range before a successful demote.
So, atomic_set() to PTRS_PER_PMD looks correct, though atomic_add() seems even safer.

author
#125 lore raw

RFC PATCH v2 23/23 KVM: TDX: Turn on PG_LEVEL_2M after TD is RUNNABLE

In reply to Yan Zhao

Turn on PG_LEVEL_2M in tdx_gmem_private_max_mapping_level() when TD is RUNNABLE.

Update the warnings and KVM_BUG_ON() info elsewhere to match that 2MB mappings are permitted after TD is RUNNABLE.

Opportunistically, remove the unused params "gfn" and "pfn" in tdx_mem_page_record_premap_cnt().

---
RFC v2:
- Merged RFC v1's patch 4 (forcing PG_LEVEL_4K before TD runnable) with patch 9 (allowing PG_LEVEL_2M after TD runnable).

---
 arch/x86/kvm/vmx/tdx.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
arch/x86/kvm/vmx/tdx.c +15 -14
index 6e061d659639..a3e1ac044ee9 100644
@@ -1633,12 +1633,11 @@ static int tdx_mem_page_aug(struct kvm *kvm, gfn_t gfn,
  * The counter has to be zero on KVM_TDX_FINALIZE_VM, to ensure that there
  * are no half-initialized shared EPT pages.
  */
-static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, gfn_t gfn,
-					  enum pg_level level, kvm_pfn_t pfn)
+static int tdx_mem_page_record_premap_cnt(struct kvm *kvm, enum pg_level level)
 {
 	struct kvm_tdx *kvm_tdx = to_kvm_tdx(kvm);
 
-	if (KVM_BUG_ON(kvm->arch.pre_fault_allowed, kvm))
+	if (KVM_BUG_ON(kvm->arch.pre_fault_allowed || level != PG_LEVEL_4K, kvm))
 		return -EINVAL;
 
 	/* nr_premapped will be decreased when tdh_mem_page_add() is called. */
@@ -1667,10 +1666,6 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,
 	if (ret)
 		return ret;
 
-	/* TODO: handle large pages. */
-	if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
-		return -EINVAL;
-
 	/*
 	 * Read 'pre_fault_allowed' before 'kvm_tdx->state'; see matching
 	 * barrier in tdx_td_finalize().
@@ -1680,7 +1675,7 @@ static int tdx_sept_set_private_spte(struct kvm *kvm, gfn_t gfn,
 	if (likely(kvm_tdx->state == TD_STATE_RUNNABLE))
 		ret = tdx_mem_page_aug(kvm, gfn, level, page);
 	else
-		ret = tdx_mem_page_record_premap_cnt(kvm, gfn, level, pfn);
+		ret = tdx_mem_page_record_premap_cnt(kvm, level);
 
 	if (ret)
 		tdx_pamt_put(page, level);
@@ -1697,8 +1692,8 @@ static int tdx_sept_drop_private_spte(struct kvm *kvm, gfn_t gfn,
 	gpa_t gpa = gfn_to_gpa(gfn);
 	u64 err, entry, level_state;
 
-	/* TODO: handle large pages. */
-	if (KVM_BUG_ON(level != PG_LEVEL_4K, kvm))
+	/* Large page is not supported before TD runnable,*/
+	if (KVM_BUG_ON(kvm_tdx->state != TD_STATE_RUNNABLE && level != PG_LEVEL_4K, kvm))
 		return -EINVAL;
 
 	if (KVM_BUG_ON(!is_hkid_assigned(kvm_tdx), kvm))
@@ -1791,7 +1786,7 @@ static int tdx_sept_link_private_spt(struct kvm *kvm, gfn_t gfn,
 static int tdx_is_sept_zap_err_due_to_premap(struct kvm_tdx *kvm_tdx, u64 err,
 					     u64 entry, int level)
 {
-	if (!err || kvm_tdx->state == TD_STATE_RUNNABLE)
+	if (!err || kvm_tdx->state == TD_STATE_RUNNABLE || level > PG_LEVEL_4K)
 		return false;
 
 	if (err != (TDX_EPT_ENTRY_STATE_INCORRECT | TDX_OPERAND_ID_RCX))
@@ -1811,8 +1806,8 @@ static int tdx_sept_zap_private_spte(struct kvm *kvm, gfn_t gfn,
 	gpa_t gpa = gfn_to_gpa(gfn) & KVM_HPAGE_MASK(level);
 	u64 err, entry, level_state;
 
-	/* For now large page isn't supported yet. */
-	WARN_ON_ONCE(level != PG_LEVEL_4K);
+	/* Large page is not supported before TD runnable,*/
+	WARN_ON_ONCE(kvm_tdx->state != TD_STATE_RUNNABLE && level != PG_LEVEL_4K);
 
 	err = tdh_mem_range_block(&kvm_tdx->td, gpa, tdx_level, &entry, &level_state);
 
@@ -1993,6 +1988,9 @@ static int tdx_sept_remove_private_spte(struct kvm *kvm, gfn_t gfn,
 	struct folio *folio = page_folio(page);
 	int ret;
 
+	WARN_ON_ONCE(folio_page_idx(folio, page) + KVM_PAGES_PER_HPAGE(level) >
+		     folio_nr_pages(folio));
+
 	if (!is_hkid_assigned(to_kvm_tdx(kvm))) {
 		KVM_BUG_ON(!kvm->vm_dead, kvm);
 
@@ -3470,7 +3468,10 @@ int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp)
 
 int tdx_gmem_private_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn)
 {
-	return PG_LEVEL_4K;
+	if (unlikely(to_kvm_tdx(kvm)->state != TD_STATE_RUNNABLE))
+		return PG_LEVEL_4K;
+
+	return PG_LEVEL_2M;
 }
 
 static int tdx_online_cpu(unsigned int cpu)
Signature · 1 line
2.43.2
#126 lore raw

Re: [RFC PATCH v2 23/23] KVM: TDX: Turn on PG_LEVEL_2M after TD is RUNNABLE

Quoted 3 lines · Yan Zhao
arch/x86/kvm/vmx/tdx.c +2 -0
+	/* Large page is not supported before TD runnable,*/
+	if (KVM_BUG_ON(kvm_tdx->state != TD_STATE_RUNNABLE && level != PG_LEVEL_4K, kvm))

  return -EINVAL;

Not a particular comment to this patch, but could you elaborate a little bit why PROMOTE isn't supported in this series? This doesn't seem to be mentioned anywhere in this series (not in the coverletter either).

E.g., theoretically, I think we can have a way to PROMOTE mappings for initial memory pages (via TDH.MEM.PAGE.ADD), e.g., right before the TD is becoming runnable?

author
#127 lore raw

Re: [RFC PATCH v2 23/23] KVM: TDX: Turn on PG_LEVEL_2M after TD is RUNNABLE

Quoted 8 lines · Kai
Quoted 3 lines · Yan Zhao
arch/x86/kvm/vmx/tdx.c +2 -0
+	/* Large page is not supported before TD runnable,*/
+	if (KVM_BUG_ON(kvm_tdx->state != TD_STATE_RUNNABLE && level != PG_LEVEL_4K, kvm))

  return -EINVAL;

Not a particular comment to this patch, but could you elaborate a little bit why PROMOTE isn't supported in this series? This doesn't seem to be mentioned anywhere in this series (not in the coverletter either).

I mentioned it briefly in the coverletter:

6. Page merging (page promotion)
Promotion is disallowed (in patch 7), because
- The current TDX module requires all 4KB leafs to be either all PENDING or all ACCEPTED before a successful promotion to 2MB. This requirement prevents successful page merging after partially converting a 2MB range from private to shared and then back to private, which is the primary scenario necessitating page promotion.
- tdh_mem_page_promote() depends on tdh_mem_range_block() in the current TDX module. Consequently, handling BUSY errors is complex, as page merging typically occurs in the fault path under a shared mmu_lock.

v1 explains it in more details (See section "Page merging (page promotion)" in [*]).

[*] https://lore.kernel.org/all/[email protected]/

Quoted 3 lines

E.g., theoretically, I think we can have a way to PROMOTE mappings for initial memory pages (via TDH.MEM.PAGE.ADD), e.g., right before the TD is becoming runnable?

Right. Kirill also asked it in in v1 [1].

Though we have no need to worry about the nr_premapped calculation after Sean's cleanup series, I think there's no need to complicate the design for the initial support, due to the limited the amount of initial memory pages.

In my environment, for a TD with 8GB memory, there are 1086 count of 2MB mapping at runtime, but the initial memory is merely 1049 4KB pages in total. So, the gain is less than 2/1000.

Will call it out in the next version.

[1] https://lore.kernel.org/all/[email protected]/

#128 lore raw

Re: [RFC PATCH v2 23/23] KVM: TDX: Turn on PG_LEVEL_2M after TD is RUNNABLE

Quoted 37 lines · Yan Zhao
Quoted 8 lines · Kai
Quoted 3 lines · Yan Zhao
arch/x86/kvm/vmx/tdx.c +2 -0
+	/* Large page is not supported before TD runnable,*/
+	if (KVM_BUG_ON(kvm_tdx->state != TD_STATE_RUNNABLE && level != PG_LEVEL_4K, kvm))

  return -EINVAL;

Not a particular comment to this patch, but could you elaborate a little bit why PROMOTE isn't supported in this series? This doesn't seem to be mentioned anywhere in this series (not in the coverletter either).

I mentioned it briefly in the coverletter:

6. Page merging (page promotion)
Promotion is disallowed (in patch 7), because
- The current TDX module requires all 4KB leafs to be either all PENDING or all ACCEPTED before a successful promotion to 2MB. This requirement prevents successful page merging after partially converting a 2MB range from private to shared and then back to private, which is the primary scenario necessitating page promotion.
- tdh_mem_page_promote() depends on tdh_mem_range_block() in the current TDX module. Consequently, handling BUSY errors is complex, as page merging typically occurs in the fault path under a shared mmu_lock.

v1 explains it in more details (See section "Page merging (page promotion)" in [*]).

[*] https://lore.kernel.org/all/[email protected]/

Quoted 3 lines

E.g., theoretically, I think we can have a way to PROMOTE mappings for initial memory pages (via TDH.MEM.PAGE.ADD), e.g., right before the TD is becoming runnable?

Right. Kirill also asked it in in v1 [1].

Though we have no need to worry about the nr_premapped calculation after Sean's cleanup series, I think there's no need to complicate the design for the initial support, due to the limited the amount of initial memory pages.

Quoted 7 lines

In my environment, for a TD with 8GB memory, there are 1086 count of 2MB mapping at runtime, but the initial memory is merely 1049 4KB pages in total. So, the gain is less than 2/1000.

Will call it out in the next version.

[1] https://lore.kernel.org/all/[email protected]/

Agreed. Thanks.