Skip to contentSkip to conversation

Dynamic PAMT

PATCH Superseded Rick Edgecombe 6 participants 41 messages Jul 18 – Jul 23

Subsystems (5)
Documentation x86 KVM KVM x86 +1 more subsystems

v7 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Rick Edgecombe

author maintainer
#0 lore raw

v7 cover Dynamic PAMT

Hi,

This is hopefully the last revision of Dynamic PAMT TDX series. Thank you to all the reviewers that helped polish off the last rough spots in v6[0]. Sean please consider acking the two KVM patches. Dave, please consider taking through tip.

Kiryl and Vishal, I left your RBs because the other changes were trivial. Please shout if you prefer to drop them.

Background
==========
Dynamic PAMT is a TDX feature that allows saving memory by allocating some of its page tracking metadata dynamically, instead of statically at boot. These static allocations take roughly 0.4% of system memory. The savings are variable depending on system and TDX usage, but could be up to 100x. For more Dynamic PAMT background, please refer to [1]. For more analysis of the savings in different scenarios, see the v6 coverleter[0].

It occurred to me that since the Dynamic PAMT effort began, RAM has become much more expensive. Consequently, this feature is even more valuable now. It would be good to enable it for TDX users.

Changes
=======
Besides the polishing type comments, there were two substantial ones. These ended up getting addressed with the same small change.

Chao asked why the TDX module doesn't do the keyid range checks itself that it requires, and then only expose the Dynamic PAMT feature0 bit when it actually can support Dynamic PAMT. It seems the TDX module is open to this change, but in any case, no modules exist today that have it. Since Dynamic PAMT enablement failure will cause TDX enablement to fail, Dynamic PAMT is made an opt-in for now by adding a kernel parameter for it. Then the kernel side keyid checks are dropped.

The other significant comment was Dave asking whether the "x86/virt/tdx:
Optimize tdx_pamt_get/put()" was really needed. Later we discussed offline to keep the patch for the sake of maintaining performance and being kind to KVM's efforts to fault under a shared lock. However, now that the feature requires an opt-in, it could be for limited use and not disturb any kernel upgraders. Then the optimization patch actually does become more optional. So here it is moved to the end. I think the patch is in good shape, but if there are any doubts we can drop it out of the initial support.

Base
====
This is based on v7.2-rc3. A full branch can be found here: [2].

Testing
=======
This series was tested in the usual suite, and also with the optimization patch removed.

[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://github.com/intel-staging/tdx/tree/dpamt_v7

Kiryl Shutsemau (9):
x86/virt/tdx: Allocate page bitmap for Dynamic PAMT x86/virt/tdx: Add tdx_alloc/free_control_page() helpers x86/virt/tdx: Allocate refcounts for Dynamic PAMT memory x86/virt/tdx: Handle multiple callers in tdx_pamt_get/put()
KVM: TDX: Allocate PAMT memory for TD and vCPU control structures
KVM: TDX: Get/put PAMT pages when (un)mapping private memory x86/virt/tdx: Enable Dynamic PAMT
Documentation/x86: Add documentation for TDX's Dynamic PAMT x86/virt/tdx: Optimize tdx_pamt_get/put()

Rick Edgecombe (2):
x86/virt/tdx: Simplify PAMT layout calculation
x86/tdx: Add APIs to support Dynamic PAMT ops from KVM's fault path

 .../admin-guide/kernel-parameters.txt         |  10 +
 Documentation/arch/x86/tdx.rst                |  28 ++
 arch/x86/include/asm/kvm-x86-ops.h            |   1 +
 arch/x86/include/asm/kvm_host.h               |   2 +
 arch/x86/include/asm/tdx.h                    |  26 +
 arch/x86/include/asm/tdx_global_metadata.h    |   3 +
 arch/x86/kvm/mmu/mmu.c                        |   4 +
 arch/x86/kvm/vmx/tdx.c                        |  98 ++--
 arch/x86/kvm/vmx/tdx.h                        |   2 +
 arch/x86/virt/vmx/tdx/tdx.c                   | 452 +++++++++++++++---
 arch/x86/virt/vmx/tdx/tdx.h                   |   2 +
 arch/x86/virt/vmx/tdx/tdx_global_metadata.c   |  23 +-
 12 files changed, 559 insertions(+), 92 deletions(-)
Signature · 1 line
2.54.0
author maintainer
#1 lore raw

PATCH v7 1/11 x86/virt/tdx: Simplify PAMT layout calculation

For each memory region that the TDX module might use (called TDMR), three separate traditional PAMT allocations are needed. There is one for each supported page size (1GB, 2MB, 4KB). These store information on each page in the TDMR. In Linux, they are allocated out of one physically contiguous block, in order to more efficiently use some internal TDX module bookkeeping resources. So some simple math is needed to break the single large allocation into three smaller allocations for each page size.

There are some commonalities in the math needed to calculate the base and size for each smaller allocation, and so an effort was made to share logic across the three. Unfortunately doing this turned out unnaturally tortured, with a loop iterating over the three page sizes, only to call into a function with case statements for each page size. In the future Dynamic PAMT will add more logic that is special to the 4KB page size, making the benefit of the math sharing even more questionable.

Three is not a very high number, so get rid of the loop and just duplicate the small calculation three times. In doing so, setup for future Dynamic PAMT changes.

Since the loop that iterates over it is gone, further simplify the code by dropping the array of intermediate size and base storage. Just store the values to their final locations. Accept the small complication of having to clear tdmr->pamt_4k_base in the error path, so that tdmr_do_pamt_func() will not try to operate on the TDMR struct when attempting to free it.

AI was used under supervision to collect/apply feedback, review code and workshop logs.

---
v7:
- Remove accidentital whitespace changes (Kiryl)
- Drop stale sentence in log (Chao)
- Better patch subject (Yan)
- Drop Assisted-by tag and cover AI use in log (Dave)

v6:
- Drop {} by moving a comment (Binbin)
- Log tweaks

---
 arch/x86/virt/vmx/tdx/tdx.c | 90 ++++++++++++-------------------------
 1 file changed, 28 insertions(+), 62 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +28 -62
index 42df8ea464c47..e77a5265c2c84 100644
@@ -514,31 +514,21 @@ static __init int fill_out_tdmrs(struct list_head *tmb_list,
  * Calculate PAMT size given a TDMR and a page size.  The returned
  * PAMT size is always aligned up to 4K page boundary.
  */
-static __init unsigned long tdmr_get_pamt_sz(struct tdmr_info *tdmr, int pgsz,
-					     u16 pamt_entry_size)
+static __init unsigned long tdmr_get_pamt_sz(struct tdmr_info *tdmr, int pgsz)
 {
 	unsigned long pamt_sz, nr_pamt_entries;
+	const int tdx_pg_size_shift[TDX_PS_NR] = { PAGE_SHIFT, PMD_SHIFT, PUD_SHIFT };
+	const u16 pamt_entry_size[TDX_PS_NR] = {
+		tdx_sysinfo.tdmr.pamt_4k_entry_size,
+		tdx_sysinfo.tdmr.pamt_2m_entry_size,
+		tdx_sysinfo.tdmr.pamt_1g_entry_size,
+	};
 
-	switch (pgsz) {
-	case TDX_PS_4K:
-		nr_pamt_entries = tdmr->size >> PAGE_SHIFT;
-		break;
-	case TDX_PS_2M:
-		nr_pamt_entries = tdmr->size >> PMD_SHIFT;
-		break;
-	case TDX_PS_1G:
-		nr_pamt_entries = tdmr->size >> PUD_SHIFT;
-		break;
-	default:
-		WARN_ON_ONCE(1);
-		return 0;
-	}
+	nr_pamt_entries = tdmr->size >> tdx_pg_size_shift[pgsz];
+	pamt_sz = nr_pamt_entries * pamt_entry_size[pgsz];
 
-	pamt_sz = nr_pamt_entries * pamt_entry_size;
 	/* TDX requires PAMT size must be 4K aligned */
-	pamt_sz = ALIGN(pamt_sz, PAGE_SIZE);
-
-	return pamt_sz;
+	return PAGE_ALIGN(pamt_sz);
 }
 
 /*
@@ -576,15 +566,11 @@ static __init int tdmr_get_nid(struct tdmr_info *tdmr, struct list_head *tmb_lis
  * within @tdmr, and set up PAMTs for @tdmr.
  */
 static __init int tdmr_set_up_pamt(struct tdmr_info *tdmr,
-				   struct list_head *tmb_list,
-				   u16 pamt_entry_size[])
+				   struct list_head *tmb_list)
 {
-	unsigned long pamt_base[TDX_PS_NR];
-	unsigned long pamt_size[TDX_PS_NR];
-	unsigned long tdmr_pamt_base;
 	unsigned long tdmr_pamt_size;
 	struct page *pamt;
-	int pgsz, nid;
+	int nid;
 
 	nid = tdmr_get_nid(tdmr, tmb_list);
 
@@ -592,12 +578,10 @@ static __init int tdmr_set_up_pamt(struct tdmr_info *tdmr,
 	 * Calculate the PAMT size for each TDX supported page size
 	 * and the total PAMT size.
 	 */
-	tdmr_pamt_size = 0;
-	for (pgsz = TDX_PS_4K; pgsz < TDX_PS_NR; pgsz++) {
-		pamt_size[pgsz] = tdmr_get_pamt_sz(tdmr, pgsz,
-					pamt_entry_size[pgsz]);
-		tdmr_pamt_size += pamt_size[pgsz];
-	}
+	tdmr->pamt_4k_size = tdmr_get_pamt_sz(tdmr, TDX_PS_4K);
+	tdmr->pamt_2m_size = tdmr_get_pamt_sz(tdmr, TDX_PS_2M);
+	tdmr->pamt_1g_size = tdmr_get_pamt_sz(tdmr, TDX_PS_1G);
+	tdmr_pamt_size = tdmr->pamt_4k_size + tdmr->pamt_2m_size + tdmr->pamt_1g_size;
 
 	/*
 	 * Allocate one chunk of physically contiguous memory for all
@@ -606,25 +590,17 @@ static __init int tdmr_set_up_pamt(struct tdmr_info *tdmr,
 	 */
 	pamt = alloc_contig_pages(tdmr_pamt_size >> PAGE_SHIFT, GFP_KERNEL,
 			nid, &node_online_map);
+
+	/*
+	 * tdmr->pamt_4k_base is still zero so the error
+	 * path of the caller will skip freeing the pamt.
+	 */
 	if (!pamt)
 		return -ENOMEM;
 
-	/*
-	 * Break the contiguous allocation back up into the
-	 * individual PAMTs for each page size.
-	 */
-	tdmr_pamt_base = page_to_pfn(pamt) << PAGE_SHIFT;
-	for (pgsz = TDX_PS_4K; pgsz < TDX_PS_NR; pgsz++) {
-		pamt_base[pgsz] = tdmr_pamt_base;
-		tdmr_pamt_base += pamt_size[pgsz];
-	}
-
-	tdmr->pamt_4k_base = pamt_base[TDX_PS_4K];
-	tdmr->pamt_4k_size = pamt_size[TDX_PS_4K];
-	tdmr->pamt_2m_base = pamt_base[TDX_PS_2M];
-	tdmr->pamt_2m_size = pamt_size[TDX_PS_2M];
-	tdmr->pamt_1g_base = pamt_base[TDX_PS_1G];
-	tdmr->pamt_1g_size = pamt_size[TDX_PS_1G];
+	tdmr->pamt_4k_base = page_to_phys(pamt);
+	tdmr->pamt_2m_base = tdmr->pamt_4k_base + tdmr->pamt_4k_size;
+	tdmr->pamt_1g_base = tdmr->pamt_2m_base + tdmr->pamt_2m_size;
 
 	return 0;
 }
@@ -655,10 +631,7 @@ static __init void tdmr_do_pamt_func(struct tdmr_info *tdmr,
 	tdmr_get_pamt(tdmr, &pamt_base, &pamt_size);
 
 	/* Do nothing if PAMT hasn't been allocated for this TDMR */
-	if (!pamt_size)
-		return;
-
-	if (WARN_ON_ONCE(!pamt_base))
+	if (!pamt_base)
 		return;
 
 	pamt_func(pamt_base, pamt_size);
@@ -684,14 +657,12 @@ static __init void tdmrs_free_pamt_all(struct tdmr_info_list *tdmr_list)
 
 /* Allocate and set up PAMTs for all TDMRs */
 static __init int tdmrs_set_up_pamt_all(struct tdmr_info_list *tdmr_list,
-					struct list_head *tmb_list,
-					u16 pamt_entry_size[])
+				 struct list_head *tmb_list)
 {
 	int i, ret = 0;
 
 	for (i = 0; i < tdmr_list->nr_consumed_tdmrs; i++) {
-		ret = tdmr_set_up_pamt(tdmr_entry(tdmr_list, i), tmb_list,
-				pamt_entry_size);
+		ret = tdmr_set_up_pamt(tdmr_entry(tdmr_list, i), tmb_list);
 		if (ret)
 			goto err;
 	}
@@ -968,18 +939,13 @@ static __init int construct_tdmrs(struct list_head *tmb_list,
 				  struct tdmr_info_list *tdmr_list,
 				  struct tdx_sys_info_tdmr *sysinfo_tdmr)
 {
-	u16 pamt_entry_size[TDX_PS_NR] = {
-		sysinfo_tdmr->pamt_4k_entry_size,
-		sysinfo_tdmr->pamt_2m_entry_size,
-		sysinfo_tdmr->pamt_1g_entry_size,
-	};
 	int ret;
 
 	ret = fill_out_tdmrs(tmb_list, tdmr_list);
 	if (ret)
 		return ret;
 
-	ret = tdmrs_set_up_pamt_all(tdmr_list, tmb_list, pamt_entry_size);
+	ret = tdmrs_set_up_pamt_all(tdmr_list, tmb_list);
 	if (ret)
 		return ret;
 
Signature · 1 line
2.54.0
author maintainer
#2 lore raw

PATCH v7 2/11 x86/virt/tdx: Allocate page bitmap for Dynamic PAMT

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

The TDX Physical Address Metadata Table (PAMT) holds data about the physical memory used by TDX, and must be allocated by the kernel during TDX module initialization.

The exact size of the required PAMT memory is determined by the TDX module and may vary between TDX module versions. Currently it is approximately 0.4% of the system memory. This is a significant commitment, especially if it is not known upfront whether the machine will run any TDX guests.

Each memory region that the TDX module might use needs three separate PAMT allocations. One for each supported page size (1GB, 2MB, 4KB). The TDX module supports a new feature designed to reduce PAMT overhead called Dynamic PAMT. Under Dynamic PAMT the 4KB level is allocated dynamically during runtime, while the 1GB and 2MB levels remain allocated on TDX module initialization.

However, in the details, Dynamic PAMT still needs some smaller per 4KB page scoped data (currently it is 1 bit per page). The TDX module exposes the number of bits as a separate piece of metadata than the 4KB static allocation for normal PAMT. Although the size is enumerated differently, it is handed to the TDX module in the same way the 4KB page size PAMT allocation is for normal PAMT.

Begin to implement Dynamic PAMT in the kernel by reading the bits-per-page needed for Dynamic PAMT. Calculate the size needed for the bitmap, and use it instead of the 4KB size determined for normal PAMT, in the case of Dynamic PAMT.

The existing metadata reading code was generated by a script, but the current plan is to stop generating this code, as the script has continued to need adjustments. So add manually written code and adjust the comment about it being autogenerated to be more generic. Start to adopt a more normal kernel code style without the ternary statements and if conditionals assignments that the auto generated code has.

AI was used under supervision to collect/apply feedback, review code and workshop logs.

---
v7:
- Re-order pamt size calculations for greater readability (Kiryl)
- Move comment to its own line in tdx_supports_dynamic_pamt() (Yan, Sohil)
- Log tweak (Sohil)
- Make comment in metadata reading more appropriate (Sohil)
- Drop Assisted-by tag and cover AI use in log (Dave)
- Move tdx_supports_dynamic_pamt() to not static inline to reduce churn in later changes

v6:
- Improve comment (Binbin)
- Log tweaks
- Mark tdmr_get_pamt_bitmap_sz() __init in response to upstream changes
- Switch to more normal kernel code style, even though it differs from the existing auto generated code.

---
 arch/x86/include/asm/tdx.h                  |  2 ++
 arch/x86/include/asm/tdx_global_metadata.h  |  3 +++
 arch/x86/virt/vmx/tdx/tdx.c                 | 29 +++++++++++++++++++--
 arch/x86/virt/vmx/tdx/tdx_global_metadata.c | 23 +++++++++++++++-
 4 files changed, 54 insertions(+), 3 deletions(-)
arch/x86/include/asm/tdx.h +2 -0
index 89e97d5761d89..d414064436221 100644
@@ -118,6 +118,8 @@ static inline bool tdx_supports_runtime_update(const struct tdx_sys_info *sysinf
 	return sysinfo->features.tdx_features0 & TDX_FEATURES0_TD_PRESERVING;
 }
 
+bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo);
+
 int tdx_guest_keyid_alloc(void);
 u32 tdx_get_nr_guest_keyids(void);
 void tdx_guest_keyid_free(unsigned int keyid);
arch/x86/include/asm/tdx_global_metadata.h +3 -0
index 41150d546589c..2a42551fc33cd 100644
@@ -21,6 +21,9 @@ struct tdx_sys_info_tdmr {
 	u16 pamt_4k_entry_size;
 	u16 pamt_2m_entry_size;
 	u16 pamt_1g_entry_size;
+
+	/* Optional metadata, if Dynamic PAMT is supported */
+	u8  pamt_page_bitmap_entry_bits;
 };
 
 struct tdx_sys_info_td_ctrl {
arch/x86/virt/vmx/tdx/tdx.c +27 -2
index e77a5265c2c84..f395f1fe95093 100644
@@ -510,6 +510,18 @@ static __init int fill_out_tdmrs(struct list_head *tmb_list,
 	return 0;
 }
 
+static __init unsigned long tdmr_get_pamt_bitmap_sz(struct tdmr_info *tdmr)
+{
+	unsigned long pamt_sz, nr_pamt_entries;
+	int bits_per_entry;
+
+	bits_per_entry = tdx_sysinfo.tdmr.pamt_page_bitmap_entry_bits;
+	nr_pamt_entries = tdmr->size >> PAGE_SHIFT;
+	pamt_sz = DIV_ROUND_UP(nr_pamt_entries * bits_per_entry, BITS_PER_BYTE);
+
+	return PAGE_ALIGN(pamt_sz);
+}
+
 /*
  * Calculate PAMT size given a TDMR and a page size.  The returned
  * PAMT size is always aligned up to 4K page boundary.
@@ -578,9 +590,16 @@ static __init int tdmr_set_up_pamt(struct tdmr_info *tdmr,
 	 * Calculate the PAMT size for each TDX supported page size
 	 * and the total PAMT size.
 	 */
-	tdmr->pamt_4k_size = tdmr_get_pamt_sz(tdmr, TDX_PS_4K);
-	tdmr->pamt_2m_size = tdmr_get_pamt_sz(tdmr, TDX_PS_2M);
 	tdmr->pamt_1g_size = tdmr_get_pamt_sz(tdmr, TDX_PS_1G);
+	tdmr->pamt_2m_size = tdmr_get_pamt_sz(tdmr, TDX_PS_2M);
+
+	if (tdx_supports_dynamic_pamt(&tdx_sysinfo)) {
+		/* With Dynamic PAMT, PAMT_4K is replaced with a bitmap */
+		tdmr->pamt_4k_size = tdmr_get_pamt_bitmap_sz(tdmr);
+	} else {
+		tdmr->pamt_4k_size = tdmr_get_pamt_sz(tdmr, TDX_PS_4K);
+	}
+
 	tdmr_pamt_size = tdmr->pamt_4k_size + tdmr->pamt_2m_size + tdmr->pamt_1g_size;
 
 	/*
@@ -1969,6 +1988,12 @@ u64 tdh_phymem_page_wbinvd_hkid(u64 hkid, kvm_pfn_t pfn)
 }
 EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid);
 
+bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo)
+{
+	/* To be enabled when kernel is ready. */
+	return false;
+}
+
 void tdx_sys_disable(void)
 {
 	struct tdx_module_args args = {};
arch/x86/virt/vmx/tdx/tdx_global_metadata.c +22 -1
index e49c300f23d43..8393d2aa59dbe 100644
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Automatically generated functions to read TDX global metadata.
+ * Functions to read TDX global metadata.
  *
  * This file doesn't compile on its own as it lacks of inclusion
  * of SEAMCALL wrapper primitive which reads global metadata.
@@ -33,6 +33,18 @@ static __init int get_tdx_sys_info_features(struct tdx_sys_info_features *sysinf
 	return ret;
 }
 
+static __init int get_tdx_sys_info_tdmr_dpamt(struct tdx_sys_info_tdmr *sysinfo_tdmr)
+{
+	int ret;
+	u64 val;
+
+	ret = read_sys_metadata_field(0x9100000100000013, &val);
+	if (!ret)
+		sysinfo_tdmr->pamt_page_bitmap_entry_bits = val;
+
+	return ret;
+}
+
 static __init int get_tdx_sys_info_tdmr(struct tdx_sys_info_tdmr *sysinfo_tdmr)
 {
 	int ret = 0;
@@ -129,5 +141,14 @@ static __init int get_tdx_sys_info(struct tdx_sys_info *sysinfo)
 	ret = ret ?: get_tdx_sys_info_td_ctrl(&sysinfo->td_ctrl);
 	ret = ret ?: get_tdx_sys_info_td_conf(&sysinfo->td_conf);
 
+	/*
+	 * The kernel supports using TDX without Dynamic PAMT, so
+	 * avoid reporting failure if it's not supported. Don't try
+	 * to support buggy TDX modules that advertise Dynamic PAMT
+	 * but don't expose the metadata.
+	 */
+	if (!ret && tdx_supports_dynamic_pamt(sysinfo))
+		ret = get_tdx_sys_info_tdmr_dpamt(&sysinfo->tdmr);
+
 	return ret;
 }
Signature · 1 line
2.54.0
author maintainer
#3 lore raw

PATCH v7 3/11 x86/virt/tdx: Add tdx_alloc/free_control_page() helpers

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add helpers to use when allocating or preparing pages that are handed to the TDX module for use as control/S-EPT pages, and thus need Dynamic PAMT adjustments.

The TDX module tracks some state for each page of physical memory that it might use. It calls this state the PAMT. It includes separate state for each page size a physical page could be utilized at within the TDX module (1GB, 2MB, 4KB). In Dynamic PAMT, only the 4KB page size state is allocated dynamically.

KVM will need to hand pages to the TDX module that it will use at 4KB granularity. So these pages will need Dynamic PAMT backing added before they are used by the TDX module, and removed afterwards.

Add tdx_alloc_control_page() and tdx_free_control_page() to handle both page allocation and Dynamic PAMT installation. Make them behave like normal alloc/free functions where allocation can fail in the case of no memory, but free (with any necessary Dynamic PAMT release) always succeeds. Do this so they can support the existing TDX flows that require teardowns to succeed.

Also create tdx_pamt_get/put() to handle installing Dynamic PAMT 4KB backing for pages that are already allocated (such as KVM's use of S-EPT page tables or guest private memory). Have them take a pfn instead of a struct page, as future changes will want to use these helpers for guest pages which are tracked by PFN.

Don't CLFLUSH the Dynamic PAMT pages handed to the TDX module, as is done for some other SEAMCALLs, as the TDX docs specify that this is only needed on "TD private memory or TD control structure page".

Since these allocations will be easily user triggerable, account the memory.

Leave logic to handle concurrency issues for future changes.

AI was used under supervision to collect/apply feedback, split patches, review code and workshop logs.

---
v7:
- Comment improvements (Chao)
- Drop unneeded addition of mm.h header include (Binbin)
- Log clarity, code style nits (Sohil)
- Drop Assisted-by tag and cover AI use in log (Dave)

v6:
The major change was to split out the concurrency stuff into a future patch. It makes it easier to explain in the log. This one is the basic functionality. Then the simple version of the concurrency and why in the next patch. Also, to get rid of the dynamically sized DPAMT backing support which was not based on a formal spec.

Details:
- Split out concurrency stuff into next patch because the log was too long
- Switch to fixed size pamt page arrays (Nikolay)
- Rename tdx_alloc_page()/tdx_free_page() to tdx_alloc_control_page()/ tdx_free_control_page() to reflect control/S-EPT purpose (Sean)
- Take gfp from the caller in tdx_alloc_control_page() (Sean)
- Narrow external API: make tdx_pamt_get()/tdx_pamt_put() static and export only tdx_alloc_control_page()/tdx_free_control_page() (note:
dropped inline helpers since the discussion on Sean's series resulted in them not being needed)
- Switch EXPORT_SYMBOL_GPL to EXPORT_SYMBOL_FOR_KVM (Sean)
- Use WARN_ON_ONCE() instead of pr_err() for TDX module failures (Sean)
- Fold alloc_pamt_array()/free_pamt_array() helpers back in and fix the error-unwind index bug (dpamt_pages[i] -> [j])
- Adjustments after struct page->pfn
- Adjustments from dropping error helper patches
- Make the free error paths more normal
- Drop gfp_t arg in tdx_alloc_control_page(). In the Sean mega v5, it was really needed because the kvm_mmu_memory_cache had a gfp_t it needed something to do with. But this was still weird because that version didn't handle allocating the DPAMT pages as the gfp_t. And in the end all the callers pass GFP_KERNEL_ACCOUNT. So just drop the arg.
- Log tweaks

---
 arch/x86/include/asm/tdx.h  |   6 ++
 arch/x86/virt/vmx/tdx/tdx.c | 163 ++++++++++++++++++++++++++++++++++++
 arch/x86/virt/vmx/tdx/tdx.h |   2 +
 3 files changed, 171 insertions(+)
arch/x86/include/asm/tdx.h +6 -0
index d414064436221..86bf37b15c705 100644
@@ -126,6 +126,12 @@ void tdx_guest_keyid_free(unsigned int keyid);
 
 void tdx_quirk_reset_paddr(unsigned long base, unsigned long size);
 
+/* Number of PAMT pages to be provided to TDX module per 2MB region of PA */
+#define TDX_DPAMT_ENTRY_PAGE_CNT 2
+
+struct page *tdx_alloc_control_page(void);
+void tdx_free_control_page(struct page *page);
+
 struct tdx_td {
 	/* TD root structure: */
 	struct page *tdr_page;
arch/x86/virt/vmx/tdx/tdx.c +163 -0

169 lines not on this page · raw message

arch/x86/virt/vmx/tdx/tdx.h +2 -0
index bdfd0e1e337ac..a886c54decaad 100644
@@ -48,6 +48,8 @@
 #define TDH_SYS_CONFIG			45
 #define TDH_SYS_SHUTDOWN		52
 #define TDH_SYS_UPDATE			53
+#define TDH_PHYMEM_PAMT_ADD		58
+#define TDH_PHYMEM_PAMT_REMOVE		59
 #define TDH_SYS_DISABLE			69
 
 /*
Signature · 1 line
2.54.0
author maintainer
#4 lore raw

PATCH v7 4/11 x86/virt/tdx: Allocate refcounts for Dynamic PAMT memory

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

The PAMT memory holds metadata for all possible TDX protected memory. Each physical address range is covered by PAMT entries at three levels (1GB, 2MB, 4KB). With Dynamic PAMT, the 4KB level of PAMT is allocated on demand. The kernel supplies the TDX module with page pairs to store the 4KB level entries, which cover 2MB of host physical memory. The kernel must provide this page pair before using pages from the range for TDX. If this is not done, SEAMCALLs that give the pages to be protected by the TDX module will fail.

Allocate reference counters for every 2MB range to track TDX memory usage. This can be used to handle concurrent get/put callers, in order to accurately determine when the dynamic 4KB level of Dynamic PAMT needs to be allocated and when it can be freed.

This allocation will currently consume 2MB for every 1TB of address space from 0 to max_pfn. The allocation size will depend on how the RAM is physically laid out. In a worst case scenario where the entire 52 bit address space is covered this would be 8GB. Then the Dynamic PAMT refcount allocations could hypothetically cause the savings from Dynamic PAMT to go negative on exotic platforms with sparse, small amounts of memory.

Future changes could reduce this refcount overhead to be only allocating refcounts for physical ranges that contain memory that TDX can use. However, this is left for future work.

AI was used under supervision to collect/apply feedback, review code and workshop logs.

---
v7:
- Annote functions __init (Chao)
- Log tweaks (Yan)
- Stanardize on memory units in the text (Sohil)
- Delete unneeded comment (Sohil)
- Use vzalloc() (Sohil)
- Drop Assisted-by tag and cover AI use in log (Dave)

v6:
- Remove confusing reference to allocating PAMT memory in pamt_refcounts comment. (Yan)
- Rename "metadata" function names that really deal with refcounts, as metadata already has a different meaning in TDX.
- Move tdx_find_pamt_refcount() to this patch to aid in reviewability

---
 arch/x86/virt/vmx/tdx/tdx.c | 53 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 52 insertions(+), 1 deletion(-)
arch/x86/virt/vmx/tdx/tdx.c +52 -1
index bfd9928c10249..b2ddd3c192645 100644
@@ -30,6 +30,7 @@
 #include <linux/suspend.h>
 #include <linux/syscore_ops.h>
 #include <linux/idr.h>
+#include <linux/vmalloc.h>
 #include <asm/page.h>
 #include <asm/special_insns.h>
 #include <asm/msr-index.h>
@@ -63,6 +64,14 @@ static DEFINE_PER_CPU(bool, tdx_lp_initialized);
 
 static struct tdmr_info_list tdx_tdmr_list;
 
+/*
+ * On a machine with Dynamic PAMT, the kernel maintains a reference counter
+ * for every 2MB range. The counter indicates how many users there are for
+ * the PAMT memory of the 2MB range. The kernel allocates PAMT refcounts at
+ * initialization.
+ */
+static atomic_t *pamt_refcounts;
+
 /* All TDX-usable memory regions.  Protected by mem_hotplug_lock. */
 static LIST_HEAD(tdx_memlist);
 
@@ -252,6 +261,42 @@ static struct syscore tdx_syscore = {
 	.ops = &tdx_syscore_ops,
 };
 
+/*
+ * Allocate PAMT reference counters for all physical memory.
+ *
+ * It consumes 2MB for every 1TB of physical memory.
+ */
+static __init int init_pamt_refcounts(void)
+{
+	size_t size = DIV_ROUND_UP(max_pfn, PTRS_PER_PTE) * sizeof(*pamt_refcounts);
+
+	if (!tdx_supports_dynamic_pamt(&tdx_sysinfo))
+		return 0;
+
+	pamt_refcounts = vzalloc(size);
+	if (!pamt_refcounts)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static __init void free_pamt_refcounts(void)
+{
+	if (!tdx_supports_dynamic_pamt(&tdx_sysinfo))
+		return;
+
+	vfree(pamt_refcounts);
+	pamt_refcounts = NULL;
+}
+
+static atomic_t * __maybe_unused tdx_find_pamt_refcount(unsigned long pfn)
+{
+	/* Find which PMD a PFN is in. */
+	unsigned long index = pfn >> (PMD_SHIFT - PAGE_SHIFT);
+
+	return &pamt_refcounts[index];
+}
+
 /*
  * Add a memory region as a TDX memory block.  The caller must make sure
  * all memory regions are added in address ascending order and don't
@@ -1152,10 +1197,14 @@ static __init int init_tdx_module(void)
 	 */
 	get_online_mems();
 
-	ret = build_tdx_memlist(&tdx_memlist);
+	ret = init_pamt_refcounts();
 	if (ret)
 		goto out_put_tdxmem;
 
+	ret = build_tdx_memlist(&tdx_memlist);
+	if (ret)
+		goto err_free_pamt_refcounts;
+
 	/* Allocate enough space for constructing TDMRs */
 	ret = alloc_tdmr_list(&tdx_tdmr_list, &tdx_sysinfo.tdmr);
 	if (ret)
@@ -1205,6 +1254,8 @@ static __init int init_tdx_module(void)
 	free_tdmr_list(&tdx_tdmr_list);
 err_free_tdxmem:
 	free_tdx_memlist(&tdx_memlist);
+err_free_pamt_refcounts:
+	free_pamt_refcounts();
 	goto out_put_tdxmem;
 }
 
Signature · 1 line
2.54.0
author maintainer
#5 lore raw

PATCH v7 5/11 x86/virt/tdx: Handle multiple callers in tdx_pamt_get/put()

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

tdx_pamt_get()/tdx_pamt_put() unconditionally add or remove Dynamic PAMT backing for the 2MB region covering the passed page. However, multiple callers can add or remove 4KB pages that fall within the same 2MB region and in that scenario only a single PAMT entry is required.

Make the helpers handle only adding/removing Dynamic PAMT backing when required, by refcounting each 2MB range. Gate the actual Dynamic PAMT add and remove on refcount transitions (0->1 and 1->0). Serialize the refcount check and SEAMCALL with a global spinlock so the read-decide-act sequence is atomic. This also avoids TDX module BUSY errors, as the Dynamic PAMT add and remove SEAMCALLs take internal TDX module locks for the 2MB ranges of the specified PFN and the PAMT page pair PFNs. So simultaneous attempts on the same 2MB ranges of the PFNs would otherwise encounter an error, which would not be handleable in the put case.

The lock is global and heavyweight. Use simple conditional logic to keep correctness obvious. This will be optimized in a later change.

The pamt_refcounts[] are atomic_t's. They do not strictly need to be because all access is protected by pamt_lock. The overhead of an atomic_t in this situation is minuscule compared to the global lock. Leave the atomic_t in place to enable future optimization with minimal churn.

AI was used under supervision to collect/apply feedback, split patches, review code and workshop logs.

---
v7:
- Convert scoped_guard() blocks to use normal spin_un/lock() for the sake of making next patches diff cleaner
- Drop __maybe_unused from tdx_find_pamt_refcount() (Binbin)
- Switch to atomic_inc_not_zero() (Dave)
- Justify use of atomic_t in log (Sohil)
- Log/comments (Yan)
- Drop Assisted-by tag and cover AI use in log (Dave)

v6:
- Split from "x86/virt/tdx: Add tdx_alloc/free_control_page() helpers"
- Return 0 instead of ret to be clearer (Binbin)
- Clarify log (Nikolay)
- Justify why the patch is not optimized in response to comments by (Nikolay)
- Move tdx_find_pamt_refcount() to faciliate patch re-order
- Adjustments from dropping error helper patches
- Log tweaks

---
 arch/x86/virt/vmx/tdx/tdx.c | 48 +++++++++++++++++++++++++++++++++----
 1 file changed, 43 insertions(+), 5 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +43 -5
index b2ddd3c192645..343d0cccc9874 100644
@@ -289,7 +289,7 @@ static __init void free_pamt_refcounts(void)
 	pamt_refcounts = NULL;
 }
 
-static atomic_t * __maybe_unused tdx_find_pamt_refcount(unsigned long pfn)
+static atomic_t *tdx_find_pamt_refcount(unsigned long pfn)
 {
 	/* Find which PMD a PFN is in. */
 	unsigned long index = pfn >> (PMD_SHIFT - PAGE_SHIFT);
@@ -2120,10 +2120,14 @@ static u64 tdh_phymem_pamt_remove(kvm_pfn_t pfn, struct page **pamt_pages)
 	return 0;
 }
 
-/* Allocate PAMT memory for the given page */
+/* Serializes adding/removing PAMT memory */
+static DEFINE_SPINLOCK(pamt_lock);
+
+/* Bump PAMT refcount for the given pfn and allocate PAMT backing if needed. */
 static int tdx_pamt_get(kvm_pfn_t pfn)
 {
 	struct page *pamt_pages[TDX_DPAMT_ENTRY_PAGE_CNT];
+	atomic_t *pamt_refcount;
 	u64 tdx_status;
 	int ret;
 
@@ -2134,29 +2138,58 @@ static int tdx_pamt_get(kvm_pfn_t pfn)
 	if (ret)
 		return ret;
 
+	pamt_refcount = tdx_find_pamt_refcount(pfn);
+
+	spin_lock(&pamt_lock);
+
+	/*
+	 * If the pamt page is already added (i.e. refcount >= 1),
+	 * then just increment the refcount.
+	 */
+	if (atomic_inc_not_zero(pamt_refcount))
+		goto out_free;
+
+	/* Try to add the pamt page and take the refcount 0->1. */
 	tdx_status = tdh_phymem_pamt_add(pfn, pamt_pages);
-	if (tdx_status != TDX_SUCCESS) {
+	if (WARN_ON_ONCE(tdx_status != TDX_SUCCESS)) {
 		ret = -EIO;
 		goto out_free;
 	}
 
+	atomic_set(pamt_refcount, 1);
+	spin_unlock(&pamt_lock);
 	return 0;
 
 out_free:
+	spin_unlock(&pamt_lock);
 	free_pamt_array(pamt_pages);
 
 	return ret;
 }
 
-/* Free PAMT memory for the given page */
+/* Drop PAMT refcount for the given pfn and free PAMT backing if needed. */
 static void tdx_pamt_put(kvm_pfn_t pfn)
 {
 	struct page *pamt_pages[TDX_DPAMT_ENTRY_PAGE_CNT] = {};
+	atomic_t *pamt_refcount;
 	u64 tdx_status;
 
 	if (!tdx_supports_dynamic_pamt(&tdx_sysinfo))
 		return;
 
+	pamt_refcount = tdx_find_pamt_refcount(pfn);
+
+	spin_lock(&pamt_lock);
+	/*
+	 * If there is more than 1 reference on the pamt page, don't
+	 * remove it yet. Just decrement the refcount.
+	 */
+	if (atomic_read(pamt_refcount) > 1) {
+		atomic_dec(pamt_refcount);
+		goto out_unlock;
+	}
+
+	/* Try to remove the pamt page and take the refcount 1->0. */
 	tdx_status = tdh_phymem_pamt_remove(pfn, pamt_pages);
 
 	/*
@@ -2167,9 +2200,14 @@ static void tdx_pamt_put(kvm_pfn_t pfn)
 	 * the dangling PAMT entry may cause future operations to fail.
 	 */
 	if (WARN_ON_ONCE(tdx_status != TDX_SUCCESS))
-		return;
+		goto out_unlock;
 
+	atomic_set(pamt_refcount, 0);
+	spin_unlock(&pamt_lock);
 	free_pamt_array(pamt_pages);
+	return;
+out_unlock:
+	spin_unlock(&pamt_lock);
 }
 
 /*
Signature · 1 line
2.54.0
Reviewed-by on v7 5/11
#6 lore raw

Re: [PATCH v7 05/11] x86/virt/tdx: Handle multiple callers in tdx_pamt_get/put()

Quoted 28 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

tdx_pamt_get()/tdx_pamt_put() unconditionally add or remove Dynamic PAMT backing for the 2MB region covering the passed page. However, multiple callers can add or remove 4KB pages that fall within the same 2MB region and in that scenario only a single PAMT entry is required.

Make the helpers handle only adding/removing Dynamic PAMT backing when required, by refcounting each 2MB range. Gate the actual Dynamic PAMT add and remove on refcount transitions (0->1 and 1->0). Serialize the refcount check and SEAMCALL with a global spinlock so the read-decide-act sequence is atomic. This also avoids TDX module BUSY errors, as the Dynamic PAMT add and remove SEAMCALLs take internal TDX module locks for the 2MB ranges of the specified PFN and the PAMT page pair PFNs. So simultaneous attempts on the same 2MB ranges of the PFNs would otherwise encounter an error, which would not be handleable in the put case.

The lock is global and heavyweight. Use simple conditional logic to keep correctness obvious. This will be optimized in a later change.

The pamt_refcounts[] are atomic_t's. They do not strictly need to be because all access is protected by pamt_lock. The overhead of an atomic_t in this situation is minuscule compared to the global lock. Leave the atomic_t in place to enable future optimization with minimal churn.

AI was used under supervision to collect/apply feedback, split patches, review code and workshop logs.

Quoted 7 lines
author maintainer
#7 lore raw

PATCH v7 6/11 KVM: TDX: Allocate PAMT memory for TD and vCPU control structures

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Use control page helpers for allocating and freeing TD control structures, such that these operations can work for Dynamic PAMT.

The TDX module tracks some state for each page of physical memory that it might use. It calls this state the PAMT. It includes separate state for each page size a physical page could be utilized at within the TDX module (1GB, 2MB, 4KB). In Dynamic PAMT, only the 4KB page size state is allocated dynamically. So the kernel must ensure PAMT backing is installed for any 4KB page being gifted to the TDX module, and must tear down the backing when all associated gifted pages are reclaimed.

TD scoped control pages (TDR, TDCS) and vCPU scoped control pages (TDVPR, TDCX) are all handed to the TDX module at 4KB page size and are therefore subject to this requirement. Replace the raw alloc_page()/__free_page() calls for these pages with tdx_alloc/free_control_page().

Switching between special Dynamic PAMT operations or normal page alloc/free operations is handled internally in tdx_alloc/free_control_page(). So don't check for Dynamic PAMT around these calls. Just call them unconditionally. Similarly, drop the NULL checks before freeing, as tdx_free_control_page() handles NULL internally.

No functional change intended when Dynamic PAMT is not in use.

[sean: handle alloc+free+reclaim in one patch]
[rick: enhance log, reviewing, rebase, with help from AI tooling]

---
v7:
- Fixup tags (Sean)
- Missing word in log (Binbin)
- Log smoothness (Yan)

---
 arch/x86/kvm/vmx/tdx.c | 35 ++++++++++++++---------------------
 1 file changed, 14 insertions(+), 21 deletions(-)
arch/x86/kvm/vmx/tdx.c +14 -21
index 989ab29b8c6fb..f6c01eab8113b 100644
@@ -362,7 +362,7 @@ static void tdx_reclaim_control_page(struct page *ctrl_page)
 	if (tdx_reclaim_page(ctrl_page))
 		return;
 
-	__free_page(ctrl_page);
+	tdx_free_control_page(ctrl_page);
 }
 
 struct tdx_flush_vp_arg {
@@ -589,7 +589,7 @@ static void tdx_reclaim_td_control_pages(struct kvm *kvm)
 
 	tdx_quirk_reset_paddr(page_to_phys(kvm_tdx->td.tdr_page), PAGE_SIZE);
 
-	__free_page(kvm_tdx->td.tdr_page);
+	tdx_free_control_page(kvm_tdx->td.tdr_page);
 	kvm_tdx->td.tdr_page = NULL;
 }
 
@@ -2459,7 +2459,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params,
 
 	ret = -ENOMEM;
 
-	tdr_page = alloc_page(GFP_KERNEL_ACCOUNT);
+	tdr_page = tdx_alloc_control_page();
 	if (!tdr_page)
 		goto free_hkid;
 
@@ -2472,7 +2472,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params,
 		goto free_tdr;
 
 	for (i = 0; i < kvm_tdx->td.tdcs_nr_pages; i++) {
-		tdcs_pages[i] = alloc_page(GFP_KERNEL_ACCOUNT);
+		tdcs_pages[i] = tdx_alloc_control_page();
 		if (!tdcs_pages[i])
 			goto free_tdcs;
 	}
@@ -2590,10 +2590,8 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params,
 teardown:
 	/* Only free pages not yet added, so start at 'i' */
 	for (; i < kvm_tdx->td.tdcs_nr_pages; i++) {
-		if (tdcs_pages[i]) {
-			__free_page(tdcs_pages[i]);
-			tdcs_pages[i] = NULL;
-		}
+		tdx_free_control_page(tdcs_pages[i]);
+		tdcs_pages[i] = NULL;
 	}
 	if (!kvm_tdx->td.tdcs_pages)
 		kfree(tdcs_pages);
@@ -2608,16 +2606,13 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params,
 	free_cpumask_var(packages);
 
 free_tdcs:
-	for (i = 0; i < kvm_tdx->td.tdcs_nr_pages; i++) {
-		if (tdcs_pages[i])
-			__free_page(tdcs_pages[i]);
-	}
+	for (i = 0; i < kvm_tdx->td.tdcs_nr_pages; i++)
+		tdx_free_control_page(tdcs_pages[i]);
 	kfree(tdcs_pages);
 	kvm_tdx->td.tdcs_pages = NULL;
 
 free_tdr:
-	if (tdr_page)
-		__free_page(tdr_page);
+	tdx_free_control_page(tdr_page);
 	kvm_tdx->td.tdr_page = NULL;
 
 free_hkid:
@@ -2947,7 +2942,7 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx)
 	int ret, i;
 	u64 err;
 
-	page = alloc_page(GFP_KERNEL_ACCOUNT);
+	page = tdx_alloc_control_page();
 	if (!page)
 		return -ENOMEM;
 	tdx->vp.tdvpr_page = page;
@@ -2967,7 +2962,7 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx)
 	}
 
 	for (i = 0; i < kvm_tdx->td.tdcx_nr_pages; i++) {
-		page = alloc_page(GFP_KERNEL_ACCOUNT);
+		page = tdx_alloc_control_page();
 		if (!page) {
 			ret = -ENOMEM;
 			goto free_tdcx;
@@ -2989,7 +2984,7 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx)
 			 * method, but the rest are freed here.
 			 */
 			for (; i < kvm_tdx->td.tdcx_nr_pages; i++) {
-				__free_page(tdx->vp.tdcx_pages[i]);
+				tdx_free_control_page(tdx->vp.tdcx_pages[i]);
 				tdx->vp.tdcx_pages[i] = NULL;
 			}
 			return -EIO;
@@ -3017,16 +3012,14 @@ static int tdx_td_vcpu_init(struct kvm_vcpu *vcpu, u64 vcpu_rcx)
 
 free_tdcx:
 	for (i = 0; i < kvm_tdx->td.tdcx_nr_pages; i++) {
-		if (tdx->vp.tdcx_pages[i])
-			__free_page(tdx->vp.tdcx_pages[i]);
+		tdx_free_control_page(tdx->vp.tdcx_pages[i]);
 		tdx->vp.tdcx_pages[i] = NULL;
 	}
 	kfree(tdx->vp.tdcx_pages);
 	tdx->vp.tdcx_pages = NULL;
 
 free_tdvpr:
-	if (tdx->vp.tdvpr_page)
-		__free_page(tdx->vp.tdvpr_page);
+	tdx_free_control_page(tdx->vp.tdvpr_page);
 	tdx->vp.tdvpr_page = NULL;
 	tdx->vp.tdvpr_pa = 0;
 
Signature · 1 line
2.54.0
#8 lore raw

Re: [PATCH v7 06/11] KVM: TDX: Allocate PAMT memory for TD and vCPU control structures

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] cpus_read_lock() is acquired while holding kvm->lock, violating the KVM lock ordering.
- [High] misc_cg_uncharge() is called unconditionally on the error path, underflowing the cgroup limit when misc_cg_try_charge() fails. --

--- Patch [6]: [PATCH v7 06/11] KVM: TDX: Allocate PAMT memory for TD and vCPU control structures --- commit 16605575e9c07ab8d7ae8c15ce1e960fccfd95db
Author: Kiryl Shutsemau <[email protected]>

KVM: TDX: Allocate PAMT memory for TD and vCPU control structures

Use control page helpers for allocating and freeing TD control structures, such that these operations can work for Dynamic PAMT.

Quoted 4 lines
arch/x86/kvm/vmx/tdx.c +0 -0
index d1af0a752e97e..528ebb3cb0389 100644

[ ... ]

arch/x86/kvm/vmx/tdx.c static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params, · line 2464

Quoted 8 lines
arch/x86/kvm/vmx/tdx.c +1 -1
@@ -2459,7 +2459,7 @@ static int __tdx_td_init(struct kvm *kvm, struct td_params *td_params,
 
 	ret = -ENOMEM;
 
-	tdr_page = alloc_page(GFP_KERNEL_ACCOUNT);
+	tdr_page = tdx_alloc_control_page();
 	if (!tdr_page)
 		goto free_hkid;

[Severity: High]
This isn't a bug introduced by this patch, but doesn't jumping to free_hkid when misc_cg_try_charge() fails result in an underflow?

If misc_cg_try_charge(MISC_CG_RES_TDX, kvm_tdx->misc_cg, 1) in arch/x86/kvm/vmx/tdx.c:__tdx_td_init() fails, it jumps to free_hkid. tdx_hkid_free() then unconditionally calls misc_cg_uncharge(), underflowing the limit.

[Severity: High]
This is a pre-existing issue, but does calling cpus_read_lock() inside __tdx_td_init() violate KVM lock ordering?

The KVM_TDX_INIT_VM ioctl acquires kvm->lock via tdx_vm_state_guard() in arch/x86/kvm/vmx/tdx.c:tdx_vm_ioctl(), and then __tdx_td_init() calls cpus_read_lock() while kvm->lock is already held. Can this create a reverse locking dependency against CPU hotplug threads?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author maintainer
#9 lore raw

Re: [PATCH v7 06/11] KVM: TDX: Allocate PAMT memory for TD and vCPU control structures

Quoted 5 lines · On Sat, 2026-07-18 at 02:03 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [High] cpus_read_lock() is acquired while holding kvm->lock, violating the KVM lock ordering.
- [High] misc_cg_uncharge() is called unconditionally on the error path, underflowing the cgroup limit when misc_cg_try_charge() fails.

Yes, it's a pre-existing issue. Binbin has a patch for it and it also has been reported before:
https://lore.kernel.org/all/[email protected]/

Reviewed-by on v7 6/11
#10 lore raw

Re: [PATCH v7 06/11] KVM: TDX: Allocate PAMT memory for TD and vCPU control structures

In reply to Rick Edgecombe
Quoted 26 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Use control page helpers for allocating and freeing TD control structures, such that these operations can work for Dynamic PAMT.

The TDX module tracks some state for each page of physical memory that it might use. It calls this state the PAMT. It includes separate state for each page size a physical page could be utilized at within the TDX module (1GB, 2MB, 4KB). In Dynamic PAMT, only the 4KB page size state is allocated dynamically. So the kernel must ensure PAMT backing is installed for any 4KB page being gifted to the TDX module, and must tear down the backing when all associated gifted pages are reclaimed.

TD scoped control pages (TDR, TDCS) and vCPU scoped control pages (TDVPR, TDCX) are all handed to the TDX module at 4KB page size and are therefore subject to this requirement. Replace the raw alloc_page()/__free_page() calls for these pages with tdx_alloc/free_control_page().

Switching between special Dynamic PAMT operations or normal page alloc/free operations is handled internally in tdx_alloc/free_control_page(). So don't check for Dynamic PAMT around these calls. Just call them unconditionally. Similarly, drop the NULL checks before freeing, as tdx_free_control_page() handles NULL internally.

No functional change intended when Dynamic PAMT is not in use.

Quoted 9 lines
[sean: handle alloc+free+reclaim in one patch]
[rick: enhance log, reviewing, rebase, with help from AI tooling]
maintainer Acked-by on v7 6/11
#11 lore raw

Re: [PATCH v7 06/11] KVM: TDX: Allocate PAMT memory for TD and vCPU control structures

In reply to Rick Edgecombe
Quoted 35 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Use control page helpers for allocating and freeing TD control structures, such that these operations can work for Dynamic PAMT.

The TDX module tracks some state for each page of physical memory that it might use. It calls this state the PAMT. It includes separate state for each page size a physical page could be utilized at within the TDX module (1GB, 2MB, 4KB). In Dynamic PAMT, only the 4KB page size state is allocated dynamically. So the kernel must ensure PAMT backing is installed for any 4KB page being gifted to the TDX module, and must tear down the backing when all associated gifted pages are reclaimed.

TD scoped control pages (TDR, TDCS) and vCPU scoped control pages (TDVPR, TDCX) are all handed to the TDX module at 4KB page size and are therefore subject to this requirement. Replace the raw alloc_page()/__free_page() calls for these pages with tdx_alloc/free_control_page().

Switching between special Dynamic PAMT operations or normal page alloc/free operations is handled internally in tdx_alloc/free_control_page(). So don't check for Dynamic PAMT around these calls. Just call them unconditionally. Similarly, drop the NULL checks before freeing, as tdx_free_control_page() handles NULL internally.

No functional change intended when Dynamic PAMT is not in use.

[sean: handle alloc+free+reclaim in one patch]
[rick: enhance log, reviewing, rebase, with help from AI tooling]
Quoted 1 line

---

author maintainer
#12 lore raw

PATCH v7 7/11 x86/tdx: Add APIs to support Dynamic PAMT ops from KVM's fault path

In reply to Rick Edgecombe

When handling an EPT violation, KVM holds a spinlock while manipulating the EPT. Before entering the spinlock it doesn't know how many EPT page tables will need to be installed or whether a huge page will be used. For this reason it allocates a worst case number of page tables that it might need as part of servicing the EPT violation.

Under Dynamic PAMT these pre-allocated pages will potentially need to have Dynamic PAMT backing pages installed for them. KVM already has helpers to manage topping up page caches before taking the MMU lock, but they cannot be passed from KVM to arch/x86 code.

The problem of how and when to install the Dynamic PAMT backing pages for the pages given to the TDX module during the fault path has had a lot of design attempts.
- Extracting KVM's MMU caches requires too much inlined code added to headers.
- A few varieties of installing Dynamic PAMT backing when allocating the S-EPT page tables. (see links)
- Using mempool_t to transfer the pages between KVM and arch/x86 doesn't work because the component is designed more around maintaining a pool of pages, rather than topping up a continually drained cache.

So don't do these as they all had various problems. Instead just create a small simple data structure to use for handing a pre-allocated list of pages between KVM and arch/x86 code. Model this on KVM's existing MMU memory caches.

Add a tdx_pamt_cache arg to tdx_pamt_get() so it can draw pages from a cache when needed. Not all Dynamic PAMT page installations will happen under spinlock, for example TD and vCPU scoped control pages. So have tdx_pamt_get() maintain the existing behavior of allocating from the page allocator when NULL is passed for the struct tdx_pamt_cache arg. This prevents excess allocations for cases where it can be avoided.

Export the new helpers for KVM.

AI was used under supervision to review code and workshop logs.

---
v7:
- Log/comment tweaks (Yan)
- Drop Assisted-by tag and cover AI use in log (Dave)

v6:
- Filled out log from Sean's series

---
 arch/x86/include/asm/tdx.h  | 17 ++++++++++
 arch/x86/virt/vmx/tdx/tdx.c | 65 +++++++++++++++++++++++++++++++++----
 2 files changed, 76 insertions(+), 6 deletions(-)
arch/x86/include/asm/tdx.h +17 -0
index 86bf37b15c705..9cbd250bbd39b 100644
@@ -120,6 +120,23 @@ static inline bool tdx_supports_runtime_update(const struct tdx_sys_info *sysinf
 
 bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo);
 
+/* Simple structure for pre-allocating Dynamic PAMT pages outside of spinlocks. */
+struct tdx_pamt_cache {
+	struct list_head page_list;
+	int cnt;
+};
+
+static inline void tdx_init_pamt_cache(struct tdx_pamt_cache *cache)
+{
+	INIT_LIST_HEAD(&cache->page_list);
+	cache->cnt = 0;
+}
+
+void tdx_free_pamt_cache(struct tdx_pamt_cache *cache);
+int tdx_topup_pamt_cache(struct tdx_pamt_cache *cache, unsigned long npages);
+int tdx_pamt_get(kvm_pfn_t pfn, struct tdx_pamt_cache *cache);
+void tdx_pamt_put(kvm_pfn_t pfn);
+
 int tdx_guest_keyid_alloc(void);
 u32 tdx_get_nr_guest_keyids(void);
 void tdx_guest_keyid_free(unsigned int keyid);
arch/x86/virt/vmx/tdx/tdx.c +59 -6

107 lines not on this page · raw message

Signature · 1 line
2.54.0
Reviewed-by on v7 7/11
#13 lore raw

Re: [PATCH v7 07/11] x86/tdx: Add APIs to support Dynamic PAMT ops from KVM's fault path

Quoted 49 lines · Rick Edgecombe

When handling an EPT violation, KVM holds a spinlock while manipulating the EPT. Before entering the spinlock it doesn't know how many EPT page tables will need to be installed or whether a huge page will be used. For this reason it allocates a worst case number of page tables that it might need as part of servicing the EPT violation.

Under Dynamic PAMT these pre-allocated pages will potentially need to have Dynamic PAMT backing pages installed for them. KVM already has helpers to manage topping up page caches before taking the MMU lock, but they cannot be passed from KVM to arch/x86 code.

The problem of how and when to install the Dynamic PAMT backing pages for the pages given to the TDX module during the fault path has had a lot of design attempts.
- Extracting KVM's MMU caches requires too much inlined code added to headers.
- A few varieties of installing Dynamic PAMT backing when allocating the S-EPT page tables. (see links)
- Using mempool_t to transfer the pages between KVM and arch/x86 doesn't work because the component is designed more around maintaining a pool of pages, rather than topping up a continually drained cache.

So don't do these as they all had various problems. Instead just create a small simple data structure to use for handing a pre-allocated list of pages between KVM and arch/x86 code. Model this on KVM's existing MMU memory caches.

Add a tdx_pamt_cache arg to tdx_pamt_get() so it can draw pages from a cache when needed. Not all Dynamic PAMT page installations will happen under spinlock, for example TD and vCPU scoped control pages. So have tdx_pamt_get() maintain the existing behavior of allocating from the page allocator when NULL is passed for the struct tdx_pamt_cache arg. This prevents excess allocations for cases where it can be avoided.

Export the new helpers for KVM.

AI was used under supervision to review code and workshop logs.

Quoted 1 line

---

author maintainer
#14 lore raw

PATCH v7 8/11 KVM: TDX: Get/put PAMT pages when (un)mapping private memory

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add Dynamic PAMT support to KVM's S-EPT MMU by "getting" a PAMT page when adding guest memory (PAGE.ADD or PAGE.AUG), and "putting" the page when removing guest memory (PAGE.REMOVE).

To access the per-vCPU PAMT caches without plumbing @vcpu throughout the TDP MMU, begrudgingly use kvm_get_running_vcpu() to get the vCPU, and bug the VM if KVM attempts to set an S-EPT leaf without an active vCPU. KVM only supports creating _new_ mappings in page (pre)fault paths, all of which require an active vCPU.

The PAMT memory holds metadata for TDX protected memory. With Dynamic PAMT, PAMT_4K is allocated on demand. The kernel supplies the TDX module with a few pages that cover 2MB of host physical memory.

Releases are balanced via tdx_pamt_put(): every control-page free goes through tdx_free_control_page(), and guest data pages are put directly on the successful tdh_mem_page_remove() path and in the tdx_mem_page_add/aug() error path.

---
v7:
- Don't do to_tdx() before NULL check for readability (Binbin, Sean)
- Fixup tags (Sean)
- Export tdx_supports_dynamic_pamt() since it's used in KVM here and no longer an static inline.
v6:
- Don't have topup op take a min param (Yan, Sean)
- Make log match style of the rest of the series
- Adjustments from dropping error helper patches

---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  2 +
 arch/x86/kvm/mmu/mmu.c             |  4 ++
 arch/x86/kvm/vmx/tdx.c             | 63 ++++++++++++++++++++++++++----
 arch/x86/kvm/vmx/tdx.h             |  2 +
 arch/x86/virt/vmx/tdx/tdx.c        |  1 +
 6 files changed, 65 insertions(+), 8 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 83dc5086138b3..588563dfe88d5 100644
@@ -98,6 +98,7 @@ KVM_X86_OP_OPTIONAL_RET0(tdp_has_smep)
 KVM_X86_OP(load_mmu_pgd)
 KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
 KVM_X86_OP_OPTIONAL(free_external_spt)
+KVM_X86_OP_OPTIONAL_RET0(topup_external_cache)
 KVM_X86_OP(has_wbinvd_exit)
 KVM_X86_OP(get_l2_tsc_offset)
 KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +2 -0
index 5f6c1ce9673b7..1c706e2d773b0 100644
@@ -1922,6 +1922,8 @@ struct kvm_x86_ops {
 	/* Update external page tables for page table about to be freed. */
 	void (*free_external_spt)(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+	int (*topup_external_cache)(struct kvm_vcpu *vcpu, int min_nr_spts);
+
 
 	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/mmu.c +4 -0
index 234d0a95abf53..6dab99654f170 100644
@@ -614,6 +614,10 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect)
 					       PT64_ROOT_MAX_LEVEL);
 		if (r)
 			return r;
+
+		r = kvm_x86_call(topup_external_cache)(vcpu, PT64_ROOT_MAX_LEVEL);
+		if (r)
+			return r;
 	}
 	r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache,
 				       PT64_ROOT_MAX_LEVEL);
arch/x86/kvm/vmx/tdx.c +55 -8

146 lines not on this page · raw message

arch/x86/kvm/vmx/tdx.h +2 -0
index ac8323a68b163..fd368e3ee060b 100644
@@ -72,6 +72,8 @@ struct vcpu_tdx {
 
 	u64 map_gpa_next;
 	u64 map_gpa_end;
+
+	struct tdx_pamt_cache pamt_cache;
 };
 
 void tdh_vp_rd_failed(struct vcpu_tdx *tdx, char *uclass, u32 field, u64 err);
arch/x86/virt/vmx/tdx/tdx.c +1 -0
index e2f11b0ba46ce..61f48fff69df6 100644
@@ -2044,6 +2044,7 @@ bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo)
 	/* To be enabled when kernel is ready. */
 	return false;
 }
+EXPORT_SYMBOL_FOR_KVM(tdx_supports_dynamic_pamt);
 
 static struct page *tdx_alloc_page_pamt_cache(struct tdx_pamt_cache *cache)
 {
Signature · 1 line
2.54.0
#15 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Infinite kernel loop in `kvm_tdp_mmu_map_private_pfn` due to permanent PAMT cache depletion on…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Infinite kernel loop in kvm_tdp_mmu_map_private_pfn due to permanent PAMT cache depletion on transient TDX module contention. --

--- Patch [8]: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory --- commit 3474807f1c1bc4dedc75e2a756b65254c986a41d
Author: Kiryl Shutsemau <[email protected]>

KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Add Dynamic PAMT support to KVM's S-EPT MMU by "getting" a PAMT page when adding guest memory (PAGE.ADD or PAGE.AUG), and "putting" the page when removing guest memory (PAGE.REMOVE).

Quoted 22 lines
arch/x86/kvm/vmx/tdx.c +6 -0
index 528ebb3cb0389..049da8559882b 100644
@@ -1679,16 +1693,28 @@ static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn,
 static int tdx_sept_map_nonleaf_spte(struct kvm *kvm, gfn_t gfn,
 				     enum pg_level level, u64 new_spte)
 {
+	struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
 	gpa_t gpa = gfn_to_gpa(gfn);
 	u64 err, entry, level_state;
 	struct page *sept_pt;
+	int ret;
+
+	if (KVM_BUG_ON(!vcpu, kvm))
+		return -EIO;
 
 	sept_pt = tdx_spte_to_sept_pt(kvm, gfn, new_spte, level);
 	if (!sept_pt)
 		return -EIO;
 
+	ret = tdx_pamt_get(page_to_pfn(sept_pt), &to_tdx(vcpu)->pamt_cache);

arch/x86/kvm/vmx/tdx.c static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn, · line 1719

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+	if (ret)
+		return ret;
+
 	err = tdh_mem_sept_add(&to_kvm_tdx(kvm)->td, gpa, level, sept_pt,
 			       &entry, &level_state);
+	if (err)
+		tdx_pamt_put(page_to_pfn(sept_pt));
+
 	if (unlikely(tdx_operand_busy(err)))
 		return -EBUSY;

[Severity: High]
Does this error path risk causing an infinite loop if the TDX module contention leads to TDX_OPERAND_BUSY?

When tdh_mem_sept_add() returns TDX_OPERAND_BUSY, tdx_pamt_put() frees the PAMT pages directly back to the buddy allocator, not back into the vCPU's PAMT cache.

If this happens during VM initialization in a retry loop like kvm_tdp_mmu_map_private_pfn(), the vCPU memory caches are topped up outside of the loop:

    r = mmu_topup_memory_caches(vcpu, false);
    if (r)
        return r;

    do {
        ...
        r = kvm_tdp_mmu_map(vcpu, &fault);
    } while (r == RET_PF_RETRY);

Since the PAMT pages are discarded rather than returned to the cache, a few busy retries will completely empty the vCPU's PAMT cache.

On the next retry, tdx_pamt_get() will return -ENOMEM because the cache is empty and it does not fall back to alloc_page() when a cache is provided.

The TDP MMU handles -ENOMEM by returning RET_PF_RETRY, but because the cache is never replenished inside the loop, the loop will infinitely fail and retry.

Could the PAMT pages be returned to the vCPU cache instead of the buddy allocator during transient failures, or should the cache top-up be moved inside the retry loop?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author maintainer
#16 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 3 lines · On Sat, 2026-07-18 at 06:10 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Infinite kernel loop in kvm_tdp_mmu_map_private_pfn due to permanent PAMT cache depletion on transient TDX module contention. --

Our internal Sashiko found this too. It's a false positive as a real bug.

Today kvm_tdp_mmu_map_private_pfn() is only called tdx_gmem_post_populate() during TD setup. It holds the heavyweight tdx_vm_state_guard which grabs vm-

Quoted 1 line

lock, kvm->slots_lock, and all vcpu->mutex. So there should be no contention

possible.

Any potential confusion is not new either, because a similar thing could happen with the external page tables.

But Yan and I were discussing that it would be a good cleanup to fix this anyway because the reason it is not a functional issue is not clear from the code. For improved readability (and quieter sashiko reports) the topup can happen inside the retry loop. Either by moving the retry loop or moving the topup.

Quoted 78 lines

--- Patch [8]: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory --- commit 3474807f1c1bc4dedc75e2a756b65254c986a41d
Author: Kiryl Shutsemau <[email protected]>

KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Add Dynamic PAMT support to KVM's S-EPT MMU by "getting" a PAMT page when adding guest memory (PAGE.ADD or PAGE.AUG), and "putting" the page when removing guest memory (PAGE.REMOVE).

Quoted 22 lines
arch/x86/kvm/vmx/tdx.c +6 -0
index 528ebb3cb0389..049da8559882b 100644
@@ -1679,16 +1693,28 @@ static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn,
 static int tdx_sept_map_nonleaf_spte(struct kvm *kvm, gfn_t gfn,
 				     enum pg_level level, u64 new_spte)
 {
+	struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
 	gpa_t gpa = gfn_to_gpa(gfn);
 	u64 err, entry, level_state;
 	struct page *sept_pt;
+	int ret;
+
+	if (KVM_BUG_ON(!vcpu, kvm))
+		return -EIO;
 
 	sept_pt = tdx_spte_to_sept_pt(kvm, gfn, new_spte, level);
 	if (!sept_pt)
 		return -EIO;
 
+	ret = tdx_pamt_get(page_to_pfn(sept_pt), &to_tdx(vcpu)->pamt_cache);

arch/x86/kvm/vmx/tdx.c static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn, · line 1719

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +6 -0
+	if (ret)
+		return ret;
+
 	err = tdh_mem_sept_add(&to_kvm_tdx(kvm)->td, gpa, level, sept_pt,
 			       &entry, &level_state);
+	if (err)
+		tdx_pamt_put(page_to_pfn(sept_pt));
+
 	if (unlikely(tdx_operand_busy(err)))
 		return -EBUSY;

[Severity: High]
Does this error path risk causing an infinite loop if the TDX module contention leads to TDX_OPERAND_BUSY?

When tdh_mem_sept_add() returns TDX_OPERAND_BUSY, tdx_pamt_put() frees the PAMT pages directly back to the buddy allocator, not back into the vCPU's PAMT cache.

If this happens during VM initialization in a retry loop like kvm_tdp_mmu_map_private_pfn(), the vCPU memory caches are topped up outside of the loop:

    r = mmu_topup_memory_caches(vcpu, false);
    if (r)
        return r;

    do {
        ...
        r = kvm_tdp_mmu_map(vcpu, &fault);
    } while (r == RET_PF_RETRY);

Since the PAMT pages are discarded rather than returned to the cache, a few busy retries will completely empty the vCPU's PAMT cache.

On the next retry, tdx_pamt_get() will return -ENOMEM because the cache is empty and it does not fall back to alloc_page() when a cache is provided.

The TDP MMU handles -ENOMEM by returning RET_PF_RETRY, but because the cache is never replenished inside the loop, the loop will infinitely fail and retry.

Could the PAMT pages be returned to the vCPU cache instead of the buddy allocator during transient failures, or should the cache top-up be moved inside the retry loop?

#17 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 12 lines · Rick P Edgecombe
Quoted 3 lines · On Sat, 2026-07-18 at 06:10 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Infinite kernel loop in kvm_tdp_mmu_map_private_pfn due to permanent PAMT cache depletion on transient TDX module contention. --

Our internal Sashiko found this too. It's a false positive as a real bug.

Today kvm_tdp_mmu_map_private_pfn() is only called tdx_gmem_post_populate() during TD setup. It holds the heavyweight tdx_vm_state_guard which grabs vm-

Quoted 1 line

lock, kvm->slots_lock, and all vcpu->mutex. So there should be no contention

possible.

Quoted 7 lines

Any potential confusion is not new either, because a similar thing could happen with the external page tables.

But Yan and I were discussing that it would be a good cleanup to fix this anyway because the reason it is not a functional issue is not clear from the code. For improved readability (and quieter sashiko reports) the topup can happen inside the retry loop. Either by moving the retry loop or moving the topup.

Hmm, yeah, I think I agree. Super duper technically, that's a fix for an existing flaw. Because very, very, VERY theoretically, the cache of page table pages could be exhausted. E.g. if some other task managed to free non-leaf page tables while mmu_lock was dropped, thus forcing kvm_tdp_mmu_map_private_pfn() to allocate from its cache over and over. In practice, that's likely impossible thanks to holding slots_lock, but given that (a) retry should be rare and (b) kvm_mmu_topup_memory_cache() is basically free if no work needs to be done, I don't see any reason to do topup outside of the retry loop.

The other thing we should address is the call to kvm_mmu_reload(). Like cache exhaustion, it *should* be impossible for the root to be invalidated/obsoleted, thanks to holding slots lock. But as evidenced by the rash of recent shadow MMU bugs, we don't always get things perfect, and lack of defense-in-depth can be *extremely* painful.

I don't think I want to just move kvm_mmu_reload() into the loop, because KVM should provide stronger guarantees with respect to the validity of the loop, versus the population of the caches. I.e. I want to WARN if the root becomes obsolete after the initial reload. And more importantly, KVM really should check the validity of the root after acquiring mmu_lock.

We can't simply call is_page_fault_stale(), because mmu_invalidate_retry_gfn() is inherently fuzzy, i.e. could get false positives, even though the pfn provided by guest_memfd is guaranteed to be valid. E.g. if shared gfns surrounding the to-be-mapped gfn are concurrently invalidated.

So, this?

	r = kvm_mmu_reload(vcpu);
	if (r)
		return r;

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		if (WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)))
			return -EIO;

		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

The other option would be to gracefully handle an obsolete root instead of WARNing, but as above, I think I prefer to WARN.

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = kvm_mmu_reload(vcpu);
		if (r)
			return r;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
		if (is_page_fault_stale(vcpu, &fault))
			continue;

		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

My only hesitation with manually checking KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is that if more checks/functionality were added to is_page_fault_stale() in the future, then we could end up missing kvm_tdp_mmu_map_private_pfn() and introduce a bug.

Maybe we can have it both ways? WARN if roots are unexpectedly made obsolete, but fully check is_page_fault_stale() and gracefully handle an obsolete root instead of effectively terminating the guest.

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = kvm_mmu_reload(vcpu);
		if (r)
			return r;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu));

		/*
		 * Snapshot the invalidation sequence counter after acquiring
		 * mmu_lock, as guest_memfd guarantees the validity of the pfn,
		 * i.e. any concurrent invalidations are guaranteed to be
		 * irrelevant.
		 */
		fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
		if (is_page_fault_stale(vcpu, &fault))
			continue;

		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);
Quoted 27 lines
Quoted 27 lines

--- Patch [8]: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory --- commit 3474807f1c1bc4dedc75e2a756b65254c986a41d
Author: Kiryl Shutsemau <[email protected]>

KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Add Dynamic PAMT support to KVM's S-EPT MMU by "getting" a PAMT page when adding guest memory (PAGE.ADD or PAGE.AUG), and "putting" the page when removing guest memory (PAGE.REMOVE).

Quoted 17 lines
arch/x86/kvm/vmx/tdx.c +5 -0
index 528ebb3cb0389..049da8559882b 100644
@@ -1679,16 +1693,28 @@ static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn,
 static int tdx_sept_map_nonleaf_spte(struct kvm *kvm, gfn_t gfn,
 				     enum pg_level level, u64 new_spte)
 {
+	struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
 	gpa_t gpa = gfn_to_gpa(gfn);
 	u64 err, entry, level_state;
 	struct page *sept_pt;
+	int ret;
+
+	if (KVM_BUG_ON(!vcpu, kvm))
+		return -EIO;
 
Quoted 6 lines
Quoted 6 lines

arch/x86/kvm/vmx/tdx.c static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn, · line 1710

Quoted 6 lines
arch/x86/kvm/vmx/tdx.c +2 -0
 	sept_pt = tdx_spte_to_sept_pt(kvm, gfn, new_spte, level);
 	if (!sept_pt)
 		return -EIO;
 
+	ret = tdx_pamt_get(page_to_pfn(sept_pt), &to_tdx(vcpu)->pamt_cache);
+	if (ret)

Shouldn't this be a KVM_BUG_ON()? To Sashiko's point about this creating an infinite fault loop, that simply shouldn't happen. The whole point of using caches for the paging assets is to guarantee success.

Quoted 45 lines
Quoted 44 lines

arch/x86/kvm/vmx/tdx.c static struct page *tdx_spte_to_sept_pt(struct kvm *kvm, gfn_t gfn, · line 1719

Quoted 9 lines
arch/x86/kvm/vmx/tdx.c +5 -0
+		return ret;
+
 	err = tdh_mem_sept_add(&to_kvm_tdx(kvm)->td, gpa, level, sept_pt,
 			       &entry, &level_state);
+	if (err)
+		tdx_pamt_put(page_to_pfn(sept_pt));
+
 	if (unlikely(tdx_operand_busy(err)))
 		return -EBUSY;

[Severity: High]
Does this error path risk causing an infinite loop if the TDX module contention leads to TDX_OPERAND_BUSY?

When tdh_mem_sept_add() returns TDX_OPERAND_BUSY, tdx_pamt_put() frees the PAMT pages directly back to the buddy allocator, not back into the vCPU's PAMT cache.

If this happens during VM initialization in a retry loop like kvm_tdp_mmu_map_private_pfn(), the vCPU memory caches are topped up outside of the loop:

    r = mmu_topup_memory_caches(vcpu, false);
    if (r)
        return r;

    do {
        ...
        r = kvm_tdp_mmu_map(vcpu, &fault);
    } while (r == RET_PF_RETRY);

Since the PAMT pages are discarded rather than returned to the cache, a few busy retries will completely empty the vCPU's PAMT cache.

On the next retry, tdx_pamt_get() will return -ENOMEM because the cache is empty and it does not fall back to alloc_page() when a cache is provided.

The TDP MMU handles -ENOMEM by returning RET_PF_RETRY, but because the cache is never replenished inside the loop, the loop will infinitely fail and retry.

Could the PAMT pages be returned to the vCPU cache instead of the buddy allocator during transient failures, or should the cache top-up be moved inside the retry loop?

author maintainer
#18 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 10 lines · Sean Christopherson

Hmm, yeah, I think I agree.  Super duper technically, that's a fix for an existing flaw.  Because very, very, VERY theoretically, the cache of page table pages could be exhausted.  E.g. if some other task managed to free non- leaf page tables while mmu_lock was dropped, thus forcing kvm_tdp_mmu_map_private_pfn() to allocate from its cache over and over.  In practice, that's likely impossible thanks to holding slots_lock, but given that (a) retry should be rare and (b) kvm_mmu_topup_memory_cache() is basically free if no work needs to be done, I don't see any reason to do topup outside of the retry loop.

Quoted 5 lines

The other thing we should address is the call to kvm_mmu_reload().  Like cache exhaustion, it *should* be impossible for the root to be invalidated/obsoleted, thanks to holding slots lock.  But as evidenced by the rash of recent shadow MMU bugs, we don't always get things perfect, and lack of defense-in-depth can be *extremely* painful.

Should we assert that we are holding slots lock then? Otherwise the reason for the warnings would be confusing. To me at least.

Quoted 31 lines

I don't think I want to just move kvm_mmu_reload() into the loop, because KVM should provide stronger guarantees with respect to the validity of the loop, versus the population of the caches.  I.e. I want to WARN if the root becomes obsolete after the initial reload.  And more importantly, KVM really should check the validity of the root after acquiring mmu_lock.

We can't simply call is_page_fault_stale(), because mmu_invalidate_retry_gfn() is inherently fuzzy, i.e. could get false positives, even though the pfn provided by guest_memfd is guaranteed to be valid.  E.g. if shared gfns surrounding the to-be-mapped gfn are concurrently invalidated.

So, this?

	r = kvm_mmu_reload(vcpu);
	if (r)
		return r;

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
Quoted 9 lines
		guard(read_lock)(&kvm->mmu_lock);

		if
(WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)))
			return -EIO;

		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

Ok. Is there something we can add to connect the slots lock to the root freeing? Like maybe a helper to encode that rule? Or better to not wrap the delicate details? A comment instead...

#19 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 11 lines · Rick P Edgecombe
Quoted 10 lines · Sean Christopherson

Hmm, yeah, I think I agree.  Super duper technically, that's a fix for an existing flaw.  Because very, very, VERY theoretically, the cache of page table pages could be exhausted.  E.g. if some other task managed to free non- leaf page tables while mmu_lock was dropped, thus forcing kvm_tdp_mmu_map_private_pfn() to allocate from its cache over and over.  In practice, that's likely impossible thanks to holding slots_lock, but given that (a) retry should be rare and (b) kvm_mmu_topup_memory_cache() is basically free if no work needs to be done, I don't see any reason to do topup outside of the retry loop.

Quoted 8 lines
Quoted 5 lines

The other thing we should address is the call to kvm_mmu_reload().  Like cache exhaustion, it *should* be impossible for the root to be invalidated/obsoleted, thanks to holding slots lock.  But as evidenced by the rash of recent shadow MMU bugs, we don't always get things perfect, and lack of defense-in-depth can be *extremely* painful.

Should we assert that we are holding slots lock then? Otherwise the reason for the warnings would be confusing. To me at least.

It's already there:

int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn)

{
	struct kvm_page_fault fault = {
		.addr = gfn_to_gpa(gfn),
		.error_code = PFERR_GUEST_FINAL_MASK | PFERR_PRIVATE_ACCESS,
		.prefetch = true,
		.is_tdp = true,
		.nx_huge_page_workaround_enabled = is_nx_huge_page_enabled(vcpu->kvm),

		.max_level = PG_LEVEL_4K,
		.req_level = PG_LEVEL_4K,
		.goal_level = PG_LEVEL_4K,
		.is_private = true,

		.gfn = gfn,
		.slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn),
		.pfn = pfn,
		.map_writable = true,
	};
	struct kvm *kvm = vcpu->kvm;
	int r;

	lockdep_assert_held(&kvm->slots_lock);  <=====
Quoted 36 lines
Quoted 36 lines

I don't think I want to just move kvm_mmu_reload() into the loop, because KVM should provide stronger guarantees with respect to the validity of the loop, versus the population of the caches.  I.e. I want to WARN if the root becomes obsolete after the initial reload.  And more importantly, KVM really should check the validity of the root after acquiring mmu_lock.

We can't simply call is_page_fault_stale(), because mmu_invalidate_retry_gfn() is inherently fuzzy, i.e. could get false positives, even though the pfn provided by guest_memfd is guaranteed to be valid.  E.g. if shared gfns surrounding the to-be-mapped gfn are concurrently invalidated.

So, this?

	r = kvm_mmu_reload(vcpu);
	if (r)
		return r;

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		if
(WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)))
			return -EIO;
Quoted 7 lines
Quoted 3 lines
		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

Ok. Is there something we can add to connect the slots lock to the root freeing? Like maybe a helper to encode that rule? Or better to not wrap the delicate details? A comment instead...

Ya, a comment.

LOL, hilarious. I just discovered a (not fully functional) patch sitting in one of my many branches that adds the is_page_fault_stale() check, with this as the changelog:

KVM: x86/mmu: Ensure page fault isn't stale/obsolete when mapping private PFN

Add a sanity check in the helper used to map private pages into a TDX guest to ensure KVM isn't attempting to map memory into an invalid/obsolete root. It _should_ be impossible for the root to be invalid, as the only flow that marks TDP MMU roots as invalid is "fast all zap", and doing a "fast zap" is mutually exclusive with populating TDX memory thanks to slots_lock (this is also why KVM doesn't retry kvm_mmu_reload()).

Note, KVM will already WARN on an invalid root if CONFIG_KVM_PROVE_MMU=y, but the check is inexpensive compared to the cost of populating memory into a TDX guest, and not having a is_page_fault_stale() check _looks_ wrong.

I'll munge that into a mini-series to add the sanity checks. No need to hold the D-PAMT series, I see this as orthogonal hardening. I'm leaning towards the "have our cake and eat it too" option as the final resting state:

        do {
                if (signal_pending(current))
                        return -EINTR;

                if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
                        return -EIO;

                r = kvm_mmu_reload(vcpu);
                if (r)
                        return r;

                r = mmu_topup_memory_caches(vcpu, false);
                if (r)
                        return r;

                cond_resched();

                guard(read_lock)(&kvm->mmu_lock);

		/* Comment goes here. */
                WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu));

                /*
                 * Snapshot the invalidation sequence counter after acquiring
                 * mmu_lock, as guest_memfd guarantees the validity of the pfn,
                 * i.e. any concurrent invalidations are guaranteed to be
                 * irrelevant.
                 */
                fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
                if (is_page_fault_stale(vcpu, &fault))
                        continue;

                r = kvm_tdp_mmu_map(vcpu, &fault);
        } while (r == RET_PF_RETRY);
author maintainer
#20 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 4 lines · Sean Christopherson
Quoted 2 lines · Rick P Edgecombe

On Wed, 2026-07-22 at 08:12 -0700, Sean Christopherson wrote:

It's already there:

Doh, yep.

Quoted 19 lines
Quoted 3 lines

Ok. Is there something we can add to connect the slots lock to the root freeing? Like maybe a helper to encode that rule? Or better to not wrap the delicate details? A comment instead...

Ya, a comment.

LOL, hilarious. I just discovered a (not fully functional) patch sitting in one of my many branches that adds the is_page_fault_stale() check, with this as the changelog:

KVM: x86/mmu: Ensure page fault isn't stale/obsolete when mapping private PFN

Add a sanity check in the helper used to map private pages into a TDX guest to ensure KVM isn't attempting to map memory into an invalid/obsolete root. It _should_ be impossible for the root to be invalid, as the only flow that marks TDP MMU roots as invalid is "fast all zap", and doing a "fast zap" is mutually exclusive with populating TDX memory thanks to slots_lock (this is also why KVM doesn't retry kvm_mmu_reload()).

Quoted 7 lines

Note, KVM will already WARN on an invalid root if CONFIG_KVM_PROVE_MMU=y, but the check is inexpensive compared to the cost of populating memory into a TDX guest, and not having a is_page_fault_stale() check _looks_ wrong.

I'll munge that into a mini-series to add the sanity checks. No need to hold the D-PAMT series, I see this as orthogonal hardening. I'm leaning towards the "have our cake and eat it too" option as the final resting state:

Sounds good to me, thanks. I think Yan was working on a patch, but hadn't finished it.

Quoted 11 lines
        do {
                if (signal_pending(current))
                        return -EINTR;

                if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
                        return -EIO;

                r = kvm_mmu_reload(vcpu);
                if (r)
                        return r;

I'm ok either way, but I'd think to make the bugs easier to surface rather than handle them. It seems at odds with how we discussed KVM_BUG_ON() in the past. A reader may get the impression that kvm_mmu_reload() inside the retry is required.

Hmm, does the new paradigm of AI finding ancient lurking bugs tilt the safety/minimalism balance?

Quoted 24 lines
                r = mmu_topup_memory_caches(vcpu, false);
                if (r)
                        return r;

                cond_resched();

                guard(read_lock)(&kvm->mmu_lock);

		/* Comment goes here. */
                WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu));

                /*
                 * Snapshot the invalidation sequence counter after acquiring
                 * mmu_lock, as guest_memfd guarantees the validity of the pfn,
                 * i.e. any concurrent invalidations are guaranteed to be
                 * irrelevant.
                 */
                fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
                if (is_page_fault_stale(vcpu, &fault))
                        continue;

                r = kvm_tdp_mmu_map(vcpu, &fault);
        } while (r == RET_PF_RETRY);
author maintainer
#21 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

In reply to Sean Christopherson
Quoted 7 lines · Sean Christopherson
Quoted 3 lines
Quoted 3 lines

Review

Quoted 3 lines
(quoted diff) +1 -0
+	ret = tdx_pamt_get(page_to_pfn(sept_pt), &to_tdx(vcpu)-
Quoted 1 line

pamt_cache);

(quoted diff) +1 -0
+	if (ret)

Shouldn't this be a KVM_BUG_ON()?  To Sashiko's point about this creating an infinite fault loop, that simply shouldn't happen.  The whole point of using caches for the paging assets is to guarantee success.

Missed this one the first time. Yes it does seem like a good KVM_BUG_ON() case. I'll update it.

#22 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

In reply to Sean Christopherson
Quoted 42 lines · Sean Christopherson
Quoted 12 lines · Rick P Edgecombe
Quoted 3 lines · On Sat, 2026-07-18 at 06:10 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Infinite kernel loop in kvm_tdp_mmu_map_private_pfn due to permanent PAMT cache depletion on transient TDX module contention. --

Our internal Sashiko found this too. It's a false positive as a real bug.

Today kvm_tdp_mmu_map_private_pfn() is only called tdx_gmem_post_populate() during TD setup. It holds the heavyweight tdx_vm_state_guard which grabs vm-

Quoted 1 line

lock, kvm->slots_lock, and all vcpu->mutex. So there should be no contention

possible.

Quoted 7 lines

Any potential confusion is not new either, because a similar thing could happen with the external page tables.

But Yan and I were discussing that it would be a good cleanup to fix this anyway because the reason it is not a functional issue is not clear from the code. For improved readability (and quieter sashiko reports) the topup can happen inside the retry loop. Either by moving the retry loop or moving the topup.

Hmm, yeah, I think I agree. Super duper technically, that's a fix for an existing flaw. Because very, very, VERY theoretically, the cache of page table pages could be exhausted. E.g. if some other task managed to free non-leaf page tables while mmu_lock was dropped, thus forcing kvm_tdp_mmu_map_private_pfn() to allocate from its cache over and over. In practice, that's likely impossible thanks to holding slots_lock, but given that (a) retry should be rare and (b) kvm_mmu_topup_memory_cache() is basically free if no work needs to be done, I don't see any reason to do topup outside of the retry loop.

The other thing we should address is the call to kvm_mmu_reload(). Like cache exhaustion, it *should* be impossible for the root to be invalidated/obsoleted, thanks to holding slots lock. But as evidenced by the rash of recent shadow MMU bugs, we don't always get things perfect, and lack of defense-in-depth can be *extremely* painful.

I don't think I want to just move kvm_mmu_reload() into the loop, because KVM should provide stronger guarantees with respect to the validity of the loop, versus the population of the caches. I.e. I want to WARN if the root becomes obsolete after the initial reload. And more importantly, KVM really should check the validity of the root after acquiring mmu_lock.

Quoted 4 lines

We can't simply call is_page_fault_stale(), because mmu_invalidate_retry_gfn() is inherently fuzzy, i.e. could get false positives, even though the pfn provided by guest_memfd is guaranteed to be valid. E.g. if shared gfns surrounding the to-be-mapped gfn are concurrently invalidated.

Past you said "No" to checking is_page_fault_stale() [*] :) [*] https://lore.kernel.org/all/[email protected]/

Quoted 90 lines

So, this?

	r = kvm_mmu_reload(vcpu);
	if (r)
		return r;

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		if (WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)))
			return -EIO;

		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

The other option would be to gracefully handle an obsolete root instead of WARNing, but as above, I think I prefer to WARN.

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = kvm_mmu_reload(vcpu);
		if (r)
			return r;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
		if (is_page_fault_stale(vcpu, &fault))
			continue;

		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

My only hesitation with manually checking KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is that if more checks/functionality were added to is_page_fault_stale() in the future, then we could end up missing kvm_tdp_mmu_map_private_pfn() and introduce a bug.

Maybe we can have it both ways? WARN if roots are unexpectedly made obsolete, but fully check is_page_fault_stale() and gracefully handle an obsolete root instead of effectively terminating the guest.

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = kvm_mmu_reload(vcpu);
		if (r)
			return r;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu));

		/*
		 * Snapshot the invalidation sequence counter after acquiring
		 * mmu_lock, as guest_memfd guarantees the validity of the pfn,
		 * i.e. any concurrent invalidations are guaranteed to be
Quoted 9 lines
		 * irrelevant.
		 */
		fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
		if (is_page_fault_stale(vcpu, &fault))
			continue;

		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

BTW, since kvm_tdp_mmu_map_private_pfn() is currently solely invoked by TDX during the TD built phase, and was introduced to avoid redundant kvm_gmem_get_pfn() calls in the gmem population path, are there any foreseeable future users of kvm_tdp_mmu_map_private_pfn()?
If not, could we simply drop the RETRY loop, given that the locks in the TDX path already guarantee that a RETRY error will never occur?"

#23 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 43 lines · Yan Zhao
Quoted 42 lines · Sean Christopherson
Quoted 12 lines · Rick P Edgecombe
Quoted 3 lines · On Sat, 2026-07-18 at 06:10 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Infinite kernel loop in kvm_tdp_mmu_map_private_pfn due to permanent PAMT cache depletion on transient TDX module contention. --

Our internal Sashiko found this too. It's a false positive as a real bug.

Today kvm_tdp_mmu_map_private_pfn() is only called tdx_gmem_post_populate() during TD setup. It holds the heavyweight tdx_vm_state_guard which grabs vm-

Quoted 1 line

lock, kvm->slots_lock, and all vcpu->mutex. So there should be no contention

possible.

Quoted 7 lines

Any potential confusion is not new either, because a similar thing could happen with the external page tables.

But Yan and I were discussing that it would be a good cleanup to fix this anyway because the reason it is not a functional issue is not clear from the code. For improved readability (and quieter sashiko reports) the topup can happen inside the retry loop. Either by moving the retry loop or moving the topup.

Hmm, yeah, I think I agree. Super duper technically, that's a fix for an existing flaw. Because very, very, VERY theoretically, the cache of page table pages could be exhausted. E.g. if some other task managed to free non-leaf page tables while mmu_lock was dropped, thus forcing kvm_tdp_mmu_map_private_pfn() to allocate from its cache over and over. In practice, that's likely impossible thanks to holding slots_lock, but given that (a) retry should be rare and (b) kvm_mmu_topup_memory_cache() is basically free if no work needs to be done, I don't see any reason to do topup outside of the retry loop.

The other thing we should address is the call to kvm_mmu_reload(). Like cache exhaustion, it *should* be impossible for the root to be invalidated/obsoleted, thanks to holding slots lock. But as evidenced by the rash of recent shadow MMU bugs, we don't always get things perfect, and lack of defense-in-depth can be *extremely* painful.

I don't think I want to just move kvm_mmu_reload() into the loop, because KVM should provide stronger guarantees with respect to the validity of the loop, versus the population of the caches. I.e. I want to WARN if the root becomes obsolete after the initial reload. And more importantly, KVM really should check the validity of the root after acquiring mmu_lock.

Quoted 6 lines
Quoted 4 lines

We can't simply call is_page_fault_stale(), because mmu_invalidate_retry_gfn() is inherently fuzzy, i.e. could get false positives, even though the pfn provided by guest_memfd is guaranteed to be valid. E.g. if shared gfns surrounding the to-be-mapped gfn are concurrently invalidated.

Past you said "No" to checking is_page_fault_stale() [*] :) [*] https://lore.kernel.org/all/[email protected]/

And my reasoning there still stands: there will be false positives, and avoiding constant false positives requries a weird mmu_seq snapshot. To be very clear, I still find the code to be gross, but unfortunately, the onslaught of recent bugs in scenarios we _thought_ were impossible has made it abundantly clear that, at least when it's not completely insane, KVM needs to effectively "fail close" when the impossible happens. I.e. take action to ensure a bad assumption in KVM can't be abused to esclate into a DoS or UAF.

Quoted 39 lines
Quoted 39 lines

My only hesitation with manually checking KVM_REQ_MMU_FREE_OBSOLETE_ROOTS is that if more checks/functionality were added to is_page_fault_stale() in the future, then we could end up missing kvm_tdp_mmu_map_private_pfn() and introduce a bug.

Maybe we can have it both ways? WARN if roots are unexpectedly made obsolete, but fully check is_page_fault_stale() and gracefully handle an obsolete root instead of effectively terminating the guest.

	do {
		if (signal_pending(current))
			return -EINTR;

		if (kvm_test_request(KVM_REQ_VM_DEAD, vcpu))
			return -EIO;

		r = kvm_mmu_reload(vcpu);
		if (r)
			return r;

		r = mmu_topup_memory_caches(vcpu, false);
		if (r)
			return r;

		cond_resched();
	
		guard(read_lock)(&kvm->mmu_lock);

		WARN_ON_ONCE(kvm_test_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu));

		/*
		 * Snapshot the invalidation sequence counter after acquiring
		 * mmu_lock, as guest_memfd guarantees the validity of the pfn,
		 * i.e. any concurrent invalidations are guaranteed to be
		 * irrelevant.
		 */
		fault.mmu_seq = vcpu->kvm->mmu_invalidate_seq;
		if (is_page_fault_stale(vcpu, &fault))
			continue;
Quoted 7 lines
Quoted 3 lines
		r = kvm_tdp_mmu_map(vcpu, &fault);
	} while (r == RET_PF_RETRY);

BTW, since kvm_tdp_mmu_map_private_pfn() is currently solely invoked by TDX during the TD built phase, and was introduced to avoid redundant kvm_gmem_get_pfn() calls in the gmem population path, are there any foreseeable future users of kvm_tdp_mmu_map_private_pfn()?

Nope, not that I know of.

Quoted 2 lines

If not, could we simply drop the RETRY loop, given that the locks in the TDX path already guarantee that a RETRY error will never occur?"

We could, but I don't think that buys us much, because we still need the is_page_fault_stale() check, or an equivalent. At that point, removing the retry loop is probably a net negative, because it could be the difference between a race resulting in a WARN but an otherwise usable VM, and an unintentional guest DoS.

#24 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

In reply to Rick Edgecombe
Quoted 111 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add Dynamic PAMT support to KVM's S-EPT MMU by "getting" a PAMT page when adding guest memory (PAGE.ADD or PAGE.AUG), and "putting" the page when removing guest memory (PAGE.REMOVE).

To access the per-vCPU PAMT caches without plumbing @vcpu throughout the TDP MMU, begrudgingly use kvm_get_running_vcpu() to get the vCPU, and bug the VM if KVM attempts to set an S-EPT leaf without an active vCPU. KVM only supports creating _new_ mappings in page (pre)fault paths, all of which require an active vCPU.

The PAMT memory holds metadata for TDX protected memory. With Dynamic PAMT, PAMT_4K is allocated on demand. The kernel supplies the TDX module with a few pages that cover 2MB of host physical memory.

Releases are balanced via tdx_pamt_put(): every control-page free goes through tdx_free_control_page(), and guest data pages are put directly on the successful tdh_mem_page_remove() path and in the tdx_mem_page_add/aug() error path.

---
v7:
- Don't do to_tdx() before NULL check for readability (Binbin, Sean)
- Fixup tags (Sean)
- Export tdx_supports_dynamic_pamt() since it's used in KVM here and no longer an static inline.
v6:
- Don't have topup op take a min param (Yan, Sean)
- Make log match style of the rest of the series
- Adjustments from dropping error helper patches

---
  arch/x86/include/asm/kvm-x86-ops.h |  1 +
  arch/x86/include/asm/kvm_host.h    |  2 +
  arch/x86/kvm/mmu/mmu.c             |  4 ++
  arch/x86/kvm/vmx/tdx.c             | 63 ++++++++++++++++++++++++++----
  arch/x86/kvm/vmx/tdx.h             |  2 +
  arch/x86/virt/vmx/tdx/tdx.c        |  1 +
  6 files changed, 65 insertions(+), 8 deletions(-)
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 83dc5086138b3..588563dfe88d5 100644
@@ -98,6 +98,7 @@ KVM_X86_OP_OPTIONAL_RET0(tdp_has_smep)
  KVM_X86_OP(load_mmu_pgd)
  KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
  KVM_X86_OP_OPTIONAL(free_external_spt)
+KVM_X86_OP_OPTIONAL_RET0(topup_external_cache)
  KVM_X86_OP(has_wbinvd_exit)
  KVM_X86_OP(get_l2_tsc_offset)
  KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +2 -0
index 5f6c1ce9673b7..1c706e2d773b0 100644
@@ -1922,6 +1922,8 @@ struct kvm_x86_ops {
  	/* Update external page tables for page table about to be freed. */
  	void (*free_external_spt)(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+	int (*topup_external_cache)(struct kvm_vcpu *vcpu, int min_nr_spts);
+
 
  	bool (*has_wbinvd_exit)(void);
 
arch/x86/kvm/mmu/mmu.c +4 -0
index 234d0a95abf53..6dab99654f170 100644
@@ -614,6 +614,10 @@ static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu, bool maybe_indirect)
  					       PT64_ROOT_MAX_LEVEL);
  		if (r)
  			return r;
+
+		r = kvm_x86_call(topup_external_cache)(vcpu, PT64_ROOT_MAX_LEVEL);
+		if (r)
+			return r;
  	}
  	r = kvm_mmu_topup_memory_cache(&vcpu->arch.mmu_shadow_page_cache,
  				       PT64_ROOT_MAX_LEVEL);
arch/x86/kvm/vmx/tdx.c +4 -0
index f6c01eab8113b..c3b1d1f056cea 100644
@@ -681,6 +681,8 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu)
  	if (!irqchip_split(vcpu->kvm))
  		return -EINVAL;
 
+	tdx_init_pamt_cache(&tdx->pamt_cache);
+
  	fpstate_set_confidential(&vcpu->arch.guest_fpu);
  	vcpu->arch.apic->guest_apic_protected = true;
  	INIT_LIST_HEAD(&tdx->vt.pi_wakeup_list);
@@ -866,6 +868,8 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu)
  	struct vcpu_tdx *tdx = to_tdx(vcpu);
  	int i;
 
+	tdx_free_pamt_cache(&tdx->pamt_cache);
+
  	if (vcpu->cpu != -1) {
  		KVM_BUG_ON(tdx->state == VCPU_TD_STATE_INITIALIZED, vcpu->kvm);
  		tdx_flush_vp_on_cpu(vcpu);

arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 1634

Quoted 10 lines
arch/x86/kvm/vmx/tdx.c +7 -0
@@ -1621,6 +1625,16 @@ void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level)
  	td_vmcs_write64(to_tdx(vcpu), SHARED_EPT_POINTER, root_hpa);
  }
 
+static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min_nr_spts)
+{
+	/*
+	 * Don't cover the root SPT, but cover a possible 4KB private
+	 * page in addition to the SPTs. So -1 to exclude the root
+	 * SPT, and +1 for the guest page cancel out.
+	 */

That comment here doesn't seem to correspond to the code in anyway. There are no +-1 adjustments.

Quoted 2 lines
arch/x86/kvm/vmx/tdx.c +2 -0
+	return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache, min_nr_spts);
+}

<snip>

author maintainer
#25 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 11 lines · Nikolay Borisov

arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 1634

Quoted 8 lines

+static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min_nr_spts)

arch/x86/kvm/vmx/tdx.c +6 -0
+{
+	/*
+	 * Don't cover the root SPT, but cover a possible 4KB private
+	 * page in addition to the SPTs. So -1 to exclude the root
+	 * SPT, and +1 for the guest page cancel out.
+	 */

That comment here doesn't seem to correspond to the code in anyway. There are no +-1 adjustments.

The +1 and -1 cancel each other out. Hmm, how about:
/*
* Don't cover the root SPT, but cover a possible 4KB private
* page in addition to the SPTs. The -1 to exclude the root
* SPT and +1 for the guest page cancel each other out. So
* make no adjustments.
*/

And thanks for the RB tags!

Quoted 2 lines
Quoted 2 lines
(quoted diff) +1 -0
+	return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache,

min_nr_spts);

#26 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 13 lines · Rick P Edgecombe
Quoted 11 lines · Nikolay Borisov

arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 1634

Quoted 8 lines

+static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min_nr_spts)

arch/x86/kvm/vmx/tdx.c +6 -0
+{
+	/*
+	 * Don't cover the root SPT, but cover a possible 4KB private
+	 * page in addition to the SPTs. So -1 to exclude the root
+	 * SPT, and +1 for the guest page cancel out.
+	 */

That comment here doesn't seem to correspond to the code in anyway. There are no +-1 adjustments.

Quoted 7 lines

The +1 and -1 cancel each other out. Hmm, how about:
/*
* Don't cover the root SPT, but cover a possible 4KB private
* page in addition to the SPTs. The -1 to exclude the root
* SPT and +1 for the guest page cancel each other out. So
* make no adjustments.
*/

Maybe put it in code? E.g.

	/*
	 * Minus one page to exclude the root SPT, but plus one page for a
	 * possible 4KiB private mapping.
	 */
	min_nr_spts += -1 + 1;

	return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache, min_nr_spts);
author maintainer
#27 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 9 lines · Sean Christopherson

Maybe put it in code?  E.g.

	/*
	 * Minus one page to exclude the root SPT, but plus one page for a
	 * possible 4KiB private mapping.
	 */
	min_nr_spts += -1 + 1;

	return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache, min_nr_spts);

Haha, yea. I felt too silly writing that code. But I'm ok. This has definitely had enough confusion over the versions to warrant something.

#28 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

In reply to Sean Christopherson
Quoted 22 lines · Sean Christopherson
Quoted 13 lines · Rick P Edgecombe
Quoted 11 lines · Nikolay Borisov

arch/x86/kvm/vmx/tdx.c void tdx_load_mmu_pgd(struct kvm_vcpu *vcpu, hpa_t root_hpa, int pgd_level) · line 1634

Quoted 8 lines

+static int tdx_topup_external_pamt_cache(struct kvm_vcpu *vcpu, int min_nr_spts)

arch/x86/kvm/vmx/tdx.c +6 -0
+{
+	/*
+	 * Don't cover the root SPT, but cover a possible 4KB private
+	 * page in addition to the SPTs. So -1 to exclude the root
+	 * SPT, and +1 for the guest page cancel out.
+	 */

That comment here doesn't seem to correspond to the code in anyway. There are no +-1 adjustments.

Quoted 7 lines

The +1 and -1 cancel each other out. Hmm, how about:
/*
* Don't cover the root SPT, but cover a possible 4KB private
* page in addition to the SPTs. The -1 to exclude the root
* SPT and +1 for the guest page cancel each other out. So
* make no adjustments.
*/

Quoted 7 lines

Maybe put it in code? E.g.

	/*
	 * Minus one page to exclude the root SPT, but plus one page for a
	 * possible 4KiB private mapping.
	 */
	min_nr_spts += -1 + 1;

Makes sense

Quoted 2 lines
	return tdx_topup_pamt_cache(&to_tdx(vcpu)->pamt_cache, min_nr_spts);
#29 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

In reply to Rick Edgecombe
Quoted 17 lines · Rick Edgecombe
arch/x86/include/asm/kvm-x86-ops.h +1 -0
index 83dc5086138b3..588563dfe88d5 100644
@@ -98,6 +98,7 @@ KVM_X86_OP_OPTIONAL_RET0(tdp_has_smep)
 KVM_X86_OP(load_mmu_pgd)
 KVM_X86_OP_OPTIONAL_RET0(set_external_spte)
 KVM_X86_OP_OPTIONAL(free_external_spt)
+KVM_X86_OP_OPTIONAL_RET0(topup_external_cache)
 KVM_X86_OP(has_wbinvd_exit)
 KVM_X86_OP(get_l2_tsc_offset)
 KVM_X86_OP(get_l2_tsc_multiplier)
arch/x86/include/asm/kvm_host.h +0 -0
index 5f6c1ce9673b7..1c706e2d773b0 100644

arch/x86/include/asm/kvm_host.h struct kvm_x86_ops { · line 1926

Quoted 5 lines
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1922,6 +1922,8 @@ struct kvm_x86_ops {
 	/* Update external page tables for page table about to be freed. */
 	void (*free_external_spt)(struct kvm *kvm, struct kvm_mmu_page *sp);
 
+	int (*topup_external_cache)(struct kvm_vcpu *vcpu, int min_nr_spts);
+

Unnecessary newline (not your fault, but it stands out here).

Quoted 2 lines
arch/x86/include/asm/kvm_host.h +0 -0
 	bool (*has_wbinvd_exit)(void);
 
author maintainer
#30 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

Quoted 5 lines · Sean Christopherson
Quoted 5 lines

index 5f6c1ce9673b7..1c706e2d773b0 100644

arch/x86/include/asm/kvm_host.h +0 -0
@@ -1922,6 +1922,8 @@ struct kvm_x86_ops {
   	/* Update external page tables for page table about to be freed. */
Quoted 8 lines
Quoted 6 lines

   void (*free_external_spt)(struct kvm *kvm, struct kvm_mmu_page *sp);

arch/x86/include/asm/kvm_host.h +1 -0
+	int (*topup_external_cache)(struct kvm_vcpu *vcpu, int

min_nr_spts);
+

Unnecessary newline (not your fault, but it stands out here).

Hmm, yea I'll move it with the other external ones. Thanks for the comments. I'll spin another version.

maintainer Acked-by on v7 8/11
#31 lore raw

Re: [PATCH v7 08/11] KVM: TDX: Get/put PAMT pages when (un)mapping private memory

In reply to Rick Edgecombe
Quoted 29 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Add Dynamic PAMT support to KVM's S-EPT MMU by "getting" a PAMT page when adding guest memory (PAGE.ADD or PAGE.AUG), and "putting" the page when removing guest memory (PAGE.REMOVE).

To access the per-vCPU PAMT caches without plumbing @vcpu throughout the TDP MMU, begrudgingly use kvm_get_running_vcpu() to get the vCPU, and bug the VM if KVM attempts to set an S-EPT leaf without an active vCPU. KVM only supports creating _new_ mappings in page (pre)fault paths, all of which require an active vCPU.

The PAMT memory holds metadata for TDX protected memory. With Dynamic PAMT, PAMT_4K is allocated on demand. The kernel supplies the TDX module with a few pages that cover 2MB of host physical memory.

Releases are balanced via tdx_pamt_put(): every control-page free goes through tdx_free_control_page(), and guest data pages are put directly on the successful tdh_mem_page_remove() path and in the tdx_mem_page_add/aug() error path.

Quoted 1 line

---

With the nits fixed,

author maintainer
#32 lore raw

PATCH v7 9/11 x86/virt/tdx: Enable Dynamic PAMT

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

The Physical Address Metadata Table (PAMT) holds TDX metadata for physical memory and must be allocated by the kernel during TDX module initialization. Dynamic PAMT is a TDX module feature that can reduce this memory use by allocating part of the PAMT dynamically.

The TDX module exposes whether Dynamic PAMT is supported via a bit in the 'features0' metadata. Unfortunately, the TDX module exposes the feature as supported even when it does not support using it with the number of keyids currently configured in the BIOS. Since no TDX modules exist today with that issue fixed, make the feature default off to prevent users from upgrading their kernel and encountering TDX erroring out when trying to enable Dynamic PAMT.

For the decision of whether to make it a boot time option and/or compile time option, consider that Dynamic PAMT's memory savings are significant enough to make it a good default configuration. That is most TDX users should want it unless they have strange keyid configurations.

The feature increases the kernel size by 2KB (when TDX is configured in the build).

All pieces are in place to enable Dynamic PAMT if it is supported and the user passes a kernel parameter.

---
v7:
- Add kernel parameter following some twists and turns, derriving originally from a comment by (Chao)

v6:
- After Nikolai pointed out that the TDX docs actually have the Dynamic PAMT pages-per-2MB region fixed at 2 instead of variable sized, I checked over the docs more closely looking for anything else that might have been missed. Spotted this 48 bit physical address bit check in the docs, so added it.

---
 .../admin-guide/kernel-parameters.txt         |  7 +++++++
 arch/x86/include/asm/tdx.h                    |  1 +
 arch/x86/virt/vmx/tdx/tdx.c                   | 21 +++++++++++++++++--
 3 files changed, 27 insertions(+), 2 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
index b5493a7f8f228..be4928489b02f 100644
@@ -1321,6 +1321,13 @@ Kernel parameters
 			The filter can be disabled or changed to another
 			driver later using sysfs.
 
+	tdx_dpamt=
+			[X86] Controls whether TDX will use Dynamic PAMT
+			to save memory, when supported.
+
+			Valid parameters: "on", "off"
+			Default: "off"
+
 	reg_file_data_sampling=
 			[X86] Controls mitigation for Register File Data
 			Sampling (RFDS) vulnerability. RFDS is a CPU
arch/x86/include/asm/tdx.h +1 -0
index 9cbd250bbd39b..7910901a7ba21 100644
@@ -36,6 +36,7 @@
 /* Bit definitions of TDX_FEATURES0 metadata field */
 #define TDX_FEATURES0_TD_PRESERVING	BIT_ULL(1)
 #define TDX_FEATURES0_NO_RBP_MOD	BIT_ULL(18)
+#define TDX_FEATURES0_DYNAMIC_PAMT	BIT_ULL(36)
 
 #ifndef __ASSEMBLER__
 
arch/x86/virt/vmx/tdx/tdx.c +19 -2
index 61f48fff69df6..f397ea17248e3 100644
@@ -47,6 +47,8 @@
 #include "seamcall_internal.h"
 #include "tdx.h"
 
+bool tdx_enable_dpamt __ro_after_init;
+
 struct tdx_module_state {
 	bool initialized;
 	bool sysinit_done;
@@ -1028,6 +1030,8 @@ static __init int construct_tdmrs(struct list_head *tmb_list,
 	return ret;
 }
 
+#define TDX_SYS_CONFIG_DYNAMIC_PAMT	BIT(16)
+
 static __init int config_tdx_module(struct tdmr_info_list *tdmr_list,
 				    u64 global_keyid)
 {
@@ -1056,6 +1060,12 @@ static __init int config_tdx_module(struct tdmr_info_list *tdmr_list,
 	args.rcx = __pa(tdmr_pa_array);
 	args.rdx = tdmr_list->nr_consumed_tdmrs;
 	args.r8 = global_keyid;
+
+	if (tdx_supports_dynamic_pamt(&tdx_sysinfo)) {
+		pr_info("Enable Dynamic PAMT\n");
+		args.r8 |= TDX_SYS_CONFIG_DYNAMIC_PAMT;
+	}
+
 	ret = seamcall_prerr(TDH_SYS_CONFIG, &args);
 
 	/* Free the array as it is not required anymore. */
@@ -2041,8 +2051,8 @@ EXPORT_SYMBOL_FOR_KVM(tdh_phymem_page_wbinvd_hkid);
 
 bool tdx_supports_dynamic_pamt(const struct tdx_sys_info *sysinfo)
 {
-	/* To be enabled when kernel is ready. */
-	return false;
+	return sysinfo->features.tdx_features0 & TDX_FEATURES0_DYNAMIC_PAMT &&
+	       tdx_enable_dpamt;
 }
 EXPORT_SYMBOL_FOR_KVM(tdx_supports_dynamic_pamt);
 
@@ -2300,6 +2310,13 @@ void tdx_free_control_page(struct page *page)
 }
 EXPORT_SYMBOL_FOR_KVM(tdx_free_control_page);
 
+static int __init tdx_dpamt_setup(char *str)
+{
+	return kstrtobool(str, &tdx_enable_dpamt) == 0;
+}
+
+__setup("tdx_dpamt=", tdx_dpamt_setup);
+
 void tdx_sys_disable(void)
 {
 	struct tdx_module_args args = {};
Signature · 1 line
2.54.0
#33 lore raw

Re: [PATCH v7 09/11] x86/virt/tdx: Enable Dynamic PAMT

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] The global variable `tdx_enable_dpamt` is defined without the `static` keyword, polluting the…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The global variable tdx_enable_dpamt is defined without the static keyword, polluting the global namespace. --

--- Patch [9]: [PATCH v7 09/11] x86/virt/tdx: Enable Dynamic PAMT --- commit a5e057035573a191a887572dacd25f3faa79112d
Author: Kiryl Shutsemau <[email protected]>

x86/virt/tdx: Enable Dynamic PAMT

This commit enables Dynamic PAMT for TDX modules to reduce memory usage during module initialization. It introduces a kernel command line parameter "tdx_dpamt" to allow users to opt-in to this feature when supported by the hardware.

arch/x86/virt/vmx/tdx/tdx.c line 50

Quoted 9 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
index e31b62f36300d..69834b0d3e602 100644
@@ -47,6 +47,8 @@
 #include "seamcall_internal.h"
 #include "tdx.h"
 
+bool tdx_enable_dpamt __ro_after_init;

[Severity: Low]
Since tdx_enable_dpamt is only used within this file, should it be marked as static? Leaving it globally exposed can trigger compiler warnings like -Wmissing-declarations when building with W=1 and unnecessarily pollutes the global namespace.

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +1 -0
+
 struct tdx_module_state {
 	bool initialized;
 	bool sysinit_done;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author maintainer
#34 lore raw

Re: [PATCH v7 09/11] x86/virt/tdx: Enable Dynamic PAMT

Quoted 15 lines · On Sat, 2026-07-18 at 01:56 +0000, [email protected] wrote:
Quoted 15 lines

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] The global variable tdx_enable_dpamt is defined without the static keyword, polluting the global namespace. --

--- Patch [9]: [PATCH v7 09/11] x86/virt/tdx: Enable Dynamic PAMT --- commit a5e057035573a191a887572dacd25f3faa79112d
Author: Kiryl Shutsemau <[email protected]>

x86/virt/tdx: Enable Dynamic PAMT

This commit enables Dynamic PAMT for TDX modules to reduce memory usage during module initialization. It introduces a kernel command line parameter "tdx_dpamt" to allow users to opt-in to this feature when supported by the hardware.

Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -0
index e31b62f36300d..69834b0d3e602 100644
@@ -47,6 +47,8 @@
   #include "seamcall_internal.h"
   #include "tdx.h"
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines


+bool tdx_enable_dpamt __ro_after_init;

[Severity: Low]
Since tdx_enable_dpamt is only used within this file, should it be marked as static? Leaving it globally exposed can trigger compiler warnings like -Wmissing-declarations when building with W=1 and unnecessarily pollutes the global namespace.

Sigh. Yes this is valid. I forgot to make it static after deciding to move tdx_supports_dynamic_pamt() out of the header instead. Then it didn't need to reference out of this file. Doh.

Seems maybe something that could be fixed up when applying if the rest of the series look ok?

author maintainer
#35 lore raw

PATCH v7 10/11 Documentation/x86: Add documentation for TDX's Dynamic PAMT

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

Expand TDX documentation to include information on the Dynamic PAMT feature.

The new section explains PAMT support in the TDX module and how Dynamic PAMT affects the kernel memory use.

AI was used under supervision to review the docs.

---
v7:
- Spell out PAMT acronym (Binbin)
- Drop Assisted-by tag and cover AI use in log (Dave)
- Add info about kernel parameter

v6:
- Add missing word (Binbin)
- Use "::" instead of ":"
- Make format of dmesg example accurate

---
 .../admin-guide/kernel-parameters.txt         |  3 ++
 Documentation/arch/x86/tdx.rst                | 28 +++++++++++++++++++
 2 files changed, 31 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +3 -0
index be4928489b02f..7c8627ce07eb4 100644
@@ -1328,6 +1328,9 @@ Kernel parameters
 			Valid parameters: "on", "off"
 			Default: "off"
 
+			For details see:
+			Documentation/arch/x86/tdx.rst
+
 	reg_file_data_sampling=
 			[X86] Controls mitigation for Register File Data
 			Sampling (RFDS) vulnerability. RFDS is a CPU
Documentation/arch/x86/tdx.rst +28 -0
index 3303499ad4c6f..a1c2309230580 100644
@@ -200,6 +200,34 @@ reflects the TCB of the currently running TDX module and therefore
 changes after an update. By contrast, TEE_TCB_SVN reflects the TCB at TD
 launch time and is not affected.
 
+Dynamic PAMT
+------------
+
+Physical Address Metadata Table (PAMT) is memory that the TDX module needs
+to keep data about each page (think like struct page). It needs to be handed
+to the TDX module for its exclusive use. For normal PAMT, this is installed
+when the TDX module is first loaded and comes to about 0.4% of system memory.
+
+Dynamic PAMT is a TDX module feature that allows VMM to allocate part of the
+PAMT as needed (the parts for tracking 4KB size pages). The other page sizes
+(1GB and 2MB) are still allocated statically at the time of TDX module
+initialization. This reduces the amount of memory that TDX uses while TDs are
+not in use.
+
+When Dynamic PAMT is in use, dmesg shows it like::
+
+  [..] virt/tdx: Enable Dynamic PAMT
+  [..] virt/tdx: 10092 KB allocated for PAMT
+  [..] virt/tdx: TDX-Module initialized
+
+Dynamic PAMT is only enabled when supported and the ``tdx_dpamt=`` kernel
+parameter is set to "on". The feature is off by default because TDX module
+internal details prevent Dynamic PAMT from working on all keyid partitioning
+configurations. When the TDX module is fixed to include these constraints in
+its enumeration of Dynamic PAMT support, kernel support can be changed to
+default on. For more information, consult the Intel TDX documentation about
+Dynamic PAMT.
+
 TDX Interaction to Other Kernel Components
 ------------------------------------------
 
Signature · 1 line
2.54.0
author maintainer
#36 lore raw

PATCH v7 11/11 x86/virt/tdx: Optimize tdx_pamt_get/put()

In reply to Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

The Dynamic PAMT get/put helpers use a global spinlock to serialize all refcount updates and SEAMCALL invocations. This gives correct behavior for concurrent callers, but leads to contention. It is especially bad from the KVM side, which is designed to allow faulting in EPT under a shared lock. With the global spinlock, not only is the lock an exclusive one, but it is for all TDs instead of just a single one.

But taking the global lock each time is actually unnecessary. Only the 0->1 and 1->0 refcount transitions actually need the lock (to pair with SEAMCALLs that actually add and remove with the Dynamic PAMT pages). The common case of incrementing or decrementing a non-zero refcount can be done locklessly.

So create a fast and slow path. Check the refcount outside the lock and only take it for the slow path (0->1 and 1->0 transitions).

On the put side make the refcount adjustment and lock taking atomic so if a 'get' happens between them, it doesn't cause the Dynamic PAMT to be freed incorrectly. On the get side there is no technique for doing the refcount adjustment and lock atomically, so check the refcount again inside the lock.

AI was used under supervision to collect/apply feedback, review code and workshop logs. It assisted in identifying/evaluating the stale conditionals for the races resolved from the atomic_dec_and_lock() change. Separate from atomic_dec_and_lock() fallout, it suggested to change atomic_inc() to atomic_set(pamt_refcount, 1) in the put error path for the sake of being more precise, which Kiryl had also suggested in the past. The model also suggested updated comments following the atomic_dec_and_lock() change based on some directed prompting. The comments were subsequently edited or further prompted for fine tuning.

---
v7:
- Drop Assisted-by tag and cover AI use in log (Dave)
- Move to end of the series
- Use atomic_inc_not_zero() in this patch inside the spin_lock(), as suggested on non-optimized patch by. (Dave)

v6:
- Fix "tdx_pamt_add()" typo to "tdx_pamt_get()" in lost-race comment
- Fix error path bug: set ret = -EIO and use WARN_ON_ONCE() instead of pr_err() for unexpected PAMT.ADD failures (Sean)
- Use "set the refcount 0->1" wording to match atomic_set() usage
- Wrap comments to 80 columns
- Switch to atomic_dec_and_lock() and remove handling of races that are no longer needed as a result. Adjust comments as appropriate. (Dave)
- Adjustments from dropping error helper patches

---
 arch/x86/virt/vmx/tdx/tdx.c | 44 ++++++++++++++++++++++++-------------
 1 file changed, 29 insertions(+), 15 deletions(-)
arch/x86/virt/vmx/tdx/tdx.c +29 -15
index f397ea17248e3..cb0f0ef0dd5f1 100644
@@ -2166,28 +2166,41 @@ int tdx_pamt_get(kvm_pfn_t pfn, struct tdx_pamt_cache *cache)
 	if (!tdx_supports_dynamic_pamt(&tdx_sysinfo))
 		return 0;
 
-	ret = alloc_pamt_array(pamt_pages, cache);
-	if (ret)
-		return ret;
-
 	pamt_refcount = tdx_find_pamt_refcount(pfn);
 
-	spin_lock(&pamt_lock);
-
 	/*
 	 * If the pamt page is already added (i.e. refcount >= 1),
 	 * then just increment the refcount.
 	 */
+	if (atomic_inc_not_zero(pamt_refcount))
+		return 0;
+
+	ret = alloc_pamt_array(pamt_pages, cache);
+	if (ret)
+		return ret;
+
+	spin_lock(&pamt_lock);
+
+	/*
+	 * Unlike tdx_pamt_put() which uses atomic_dec_and_lock() to
+	 * atomically handle the 1->0 transition, the get side has no
+	 * equivalent combined primitive for 0->1. Recheck under the
+	 * lock since another get may have already done the 0->1
+	 * transition after both saw atomic_inc_not_zero() fail.
+	 */
 	if (atomic_inc_not_zero(pamt_refcount))
 		goto out_free;
 
-	/* Try to add the pamt page and take the refcount 0->1. */
 	tdx_status = tdh_phymem_pamt_add(pfn, pamt_pages);
 	if (WARN_ON_ONCE(tdx_status != TDX_SUCCESS)) {
 		ret = -EIO;
 		goto out_free;
 	}
 
+	/*
+	 * The refcount is zero, and this locked path is the
+	 * only way to increase it from 0->1.
+	 */
 	atomic_set(pamt_refcount, 1);
 	spin_unlock(&pamt_lock);
 	return 0;
@@ -2212,17 +2225,13 @@ void tdx_pamt_put(kvm_pfn_t pfn)
 
 	pamt_refcount = tdx_find_pamt_refcount(pfn);
 
-	spin_lock(&pamt_lock);
 	/*
 	 * If there is more than 1 reference on the pamt page, don't
 	 * remove it yet. Just decrement the refcount.
 	 */
-	if (atomic_read(pamt_refcount) > 1) {
-		atomic_dec(pamt_refcount);
-		goto out_unlock;
-	}
+	if (!atomic_dec_and_lock(pamt_refcount, &pamt_lock))
+		return;
 
-	/* Try to remove the pamt page and take the refcount 1->0. */
 	tdx_status = tdh_phymem_pamt_remove(pfn, pamt_pages);
 
 	/*
@@ -2232,10 +2241,15 @@ void tdx_pamt_put(kvm_pfn_t pfn)
 	 * failure indicates a kernel bug, memory is being leaked, and
 	 * the dangling PAMT entry may cause future operations to fail.
 	 */
-	if (WARN_ON_ONCE(tdx_status != TDX_SUCCESS))
+	if (WARN_ON_ONCE(tdx_status != TDX_SUCCESS)) {
+		/*
+		 * atomic_dec_and_lock() already decremented it to 0,
+		 * but the PAMT entry still exists since REMOVE failed.
+		 */
+		atomic_set(pamt_refcount, 1);
 		goto out_unlock;
+	}
 
-	atomic_set(pamt_refcount, 0);
 	spin_unlock(&pamt_lock);
 	free_pamt_array(pamt_pages);
 	return;
Signature · 1 line
2.54.0
#37 lore raw

Re: [PATCH v7 11/11] x86/virt/tdx: Optimize tdx_pamt_get/put()

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Using `atomic_dec_and_lock` in `tdx_pamt_put` lacks underflow protection, causing an unbalanced…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Using atomic_dec_and_lock in tdx_pamt_put lacks underflow protection, causing an unbalanced put (refcount 0) to silently underflow the refcount to -1, which breaks future tdx_pamt_get calls for that PFN. --

--- Patch [11]: [PATCH v7 11/11] x86/virt/tdx: Optimize tdx_pamt_get/put() --- commit f3bde8ccf9659f8b1e1cedb9b81df0b3b209a9fa
Author: Kiryl Shutsemau <[email protected]>

x86/virt/tdx: Optimize tdx_pamt_get/put()

The Dynamic PAMT get/put helpers use a global spinlock to serialize all refcount updates and SEAMCALL invocations. This gives correct behavior for concurrent callers, but leads to contention.

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -0
index 69834b0d3e602..4394bba7c9a7c 100644

[ ... ]

Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -1
@@ -2214,17 +2227,13 @@ void tdx_pamt_put(kvm_pfn_t pfn)
 
 	pamt_refcount = tdx_find_pamt_refcount(pfn);
 
-	spin_lock(&pamt_lock);

arch/x86/virt/vmx/tdx/tdx.c void tdx_pamt_put(kvm_pfn_t pfn) · line 2235

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +2 -4
 	/*
 	 * If there is more than 1 reference on the pamt page, don't
 	 * remove it yet. Just decrement the refcount.
 	 */
-	if (atomic_read(pamt_refcount) > 1) {
-		atomic_dec(pamt_refcount);
-		goto out_unlock;
-	}
+	if (!atomic_dec_and_lock(pamt_refcount, &pamt_lock))
+		return;

[Severity: Medium]
If an unbalanced put occurs on an un-added PAMT page (where the refcount is 0), does atomic_dec_and_lock() lack underflow protection and silently drop the refcount to -1 instead of proceeding to the SEAMCALL failure warning as the previous logic did?

lib/dec_and_lock.c:atomic_dec_and_lock() {
    /* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
    if (atomic_add_unless(atomic, -1, 1))
        return 0;
    ...
}

If the refcount silently becomes -1, would a subsequent call to tdx_pamt_get() for this PFN incorrectly succeed without adding the PAMT memory?

arch/x86/virt/vmx/tdx/tdx.c:tdx_pamt_get() {
    ...
    if (atomic_inc_not_zero(pamt_refcount))
        return 0;
    ...
}

Since -1 is not zero, atomic_inc_not_zero() would increment the refcount to 0 and return true, bypassing the tdh_phymem_pamt_add() SEAMCALL.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author maintainer
#38 lore raw

Re: [PATCH v7 11/11] x86/virt/tdx: Optimize tdx_pamt_get/put()

Quoted 2 lines · On Sat, 2026-07-18 at 02:00 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Using atomic_dec_and_lock in tdx_pamt_put lacks underflow protection, causing an unbalanced put (refcount 0) to silently underflow the refcount to -1, which breaks future tdx_pamt_get calls for that PFN.

The report is essentially that this patch doesn't handle unbalanced tdx_pamt_put() calls (a tdx_pamt_put() without a previous paired tdx_pamt_get()). So this isn't a bug per-se, but it is true that the optimization patch has a bit worse warnings in the case of buggy calling code.

More analysis... The function is:

int atomic_dec_and_lock(atomic_t *atomic, spinlock_t *lock)

{
	/* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
	if (atomic_add_unless(atomic, -1, 1))
		return 0;

	/* Otherwise do it the slow way */
	spin_lock(lock);
	if (atomic_dec_and_test(atomic))
		return 1;
	spin_unlock(lock);
	return 0;
}

In the fast path atomic_add_unless() subtracts 1 unless the counter was exactly
1. So if it was 0 (no paired get has been called), the counter would underflow.

In the non-optimized patch, the put code was more robust to this scenario. In that code if a put was done with the refcount at 0, the put would see it was less than 1 and try to remove it. The result would be a warning from the SEAMCALL failure.

So this patch trades some warnings for performance. But the warning loss is pretty small. Future buggy code is unlikely to get away without a warning. Missing PAMT is not like double free or bug like that. With the count unbalanced, then next tdx_pamt_get() will fail to add the Dynamic PAMT page pair and the TDX module will return an error when it tries to use the page.

Getting a warning exactly at the point of the unpaired put would be better. But this version has other clear benefits. On balance, I'd prefer this one. It might be possible to utilize refcount_t to get some better underflow warnings, but probably for the cost of more memory ordering things to consider.

Hmm, another cheap, but not complete option would be to just warn on get/put out paths if any strange refcounts are seen. Since the point is just to catch the error closer to where it happens. It's just developer help. Simpler code with less well timed warnings seems like a good tradeoff for developers?

Quoted 62 lines

--

--- Patch [11]: [PATCH v7 11/11] x86/virt/tdx: Optimize tdx_pamt_get/put() --- commit f3bde8ccf9659f8b1e1cedb9b81df0b3b209a9fa
Author: Kiryl Shutsemau <[email protected]>

x86/virt/tdx: Optimize tdx_pamt_get/put()

The Dynamic PAMT get/put helpers use a global spinlock to serialize all refcount updates and SEAMCALL invocations. This gives correct behavior for concurrent callers, but leads to contention.

Quoted 4 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -0
index 69834b0d3e602..4394bba7c9a7c 100644

[ ... ]

Quoted 5 lines
arch/x86/virt/vmx/tdx/tdx.c +0 -1
@@ -2214,17 +2227,13 @@ void tdx_pamt_put(kvm_pfn_t pfn)
 
 	pamt_refcount = tdx_find_pamt_refcount(pfn);
 
-	spin_lock(&pamt_lock);

arch/x86/virt/vmx/tdx/tdx.c void tdx_pamt_put(kvm_pfn_t pfn) · line 2235

Quoted 10 lines
arch/x86/virt/vmx/tdx/tdx.c +2 -4
 	/*
 	 * If there is more than 1 reference on the pamt page, don't
 	 * remove it yet. Just decrement the refcount.
 	 */
-	if (atomic_read(pamt_refcount) > 1) {
-		atomic_dec(pamt_refcount);
-		goto out_unlock;
-	}
+	if (!atomic_dec_and_lock(pamt_refcount, &pamt_lock))
+		return;

[Severity: Medium]
If an unbalanced put occurs on an un-added PAMT page (where the refcount is 0), does atomic_dec_and_lock() lack underflow protection and silently drop the refcount to -1 instead of proceeding to the SEAMCALL failure warning as the previous logic did?

lib/dec_and_lock.c:atomic_dec_and_lock() {
    /* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
    if (atomic_add_unless(atomic, -1, 1))
        return 0;
    ...
}

If the refcount silently becomes -1, would a subsequent call to tdx_pamt_get() for this PFN incorrectly succeed without adding the PAMT memory?

arch/x86/virt/vmx/tdx/tdx.c:tdx_pamt_get() {
    ...
    if (atomic_inc_not_zero(pamt_refcount))
        return 0;
    ...
}

Since -1 is not zero, atomic_inc_not_zero() would increment the refcount to 0 and return true, bypassing the tdh_phymem_pamt_add() SEAMCALL.

Reviewed-by on v7 11/11
#39 lore raw

Re: [PATCH v7 11/11] x86/virt/tdx: Optimize tdx_pamt_get/put()

In reply to Rick Edgecombe
Quoted 34 lines · Rick Edgecombe

From: "Kirill A. Shutemov" <[email protected]>

The Dynamic PAMT get/put helpers use a global spinlock to serialize all refcount updates and SEAMCALL invocations. This gives correct behavior for concurrent callers, but leads to contention. It is especially bad from the KVM side, which is designed to allow faulting in EPT under a shared lock. With the global spinlock, not only is the lock an exclusive one, but it is for all TDs instead of just a single one.

But taking the global lock each time is actually unnecessary. Only the 0->1 and 1->0 refcount transitions actually need the lock (to pair with SEAMCALLs that actually add and remove with the Dynamic PAMT pages). The common case of incrementing or decrementing a non-zero refcount can be done locklessly.

So create a fast and slow path. Check the refcount outside the lock and only take it for the slow path (0->1 and 1->0 transitions).

On the put side make the refcount adjustment and lock taking atomic so if a 'get' happens between them, it doesn't cause the Dynamic PAMT to be freed incorrectly. On the get side there is no technique for doing the refcount adjustment and lock atomically, so check the refcount again inside the lock.

AI was used under supervision to collect/apply feedback, review code and workshop logs. It assisted in identifying/evaluating the stale conditionals for the races resolved from the atomic_dec_and_lock() change. Separate from atomic_dec_and_lock() fallout, it suggested to change atomic_inc() to atomic_set(pamt_refcount, 1) in the put error path for the sake of being more precise, which Kiryl had also suggested in the past. The model also suggested updated comments following the atomic_dec_and_lock() change based on some directed prompting. The comments were subsequently edited or further prompted for fine tuning.

Quoted 5 lines
Acked-by on the v7 series
#40 lore raw
In reply to Rick Edgecombe
Quoted 11 lines · Rick Edgecombe

Kiryl Shutsemau (9):
x86/virt/tdx: Allocate page bitmap for Dynamic PAMT x86/virt/tdx: Add tdx_alloc/free_control_page() helpers x86/virt/tdx: Allocate refcounts for Dynamic PAMT memory x86/virt/tdx: Handle multiple callers in tdx_pamt_get/put()
KVM: TDX: Allocate PAMT memory for TD and vCPU control structures
KVM: TDX: Get/put PAMT pages when (un)mapping private memory x86/virt/tdx: Enable Dynamic PAMT
Documentation/x86: Add documentation for TDX's Dynamic PAMT x86/virt/tdx: Optimize tdx_pamt_get/put()

Quoted 4 lines

Rick Edgecombe (2):
x86/virt/tdx: Simplify PAMT layout calculation
x86/tdx: Add APIs to support Dynamic PAMT ops from KVM's fault path

I primarily reviewed the commit logs and assessed the implementation. The patches look good to me and are clean overall.